Good day! In my previous articles, I explained how to scrape data and interact with the DOM and shadow roots. You can find the first and second parts by following the links.
Today, I’ll show you how to interact with closed shadow roots. As you may know, shadow roots can be either open or closed. Open shadow roots allow direct access via queries, but closed shadow roots do not. However, you can still interact with them—let me show you how.
Base implementation
I won’t waste your time explaining the basic functionality—that's covered in the first part. Let’s jump straight into the base code:
static async Task InteractionWithClosedShadowRoot()
{
const int port = 9222;
var chromePath = "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome";
var userDataDir = Path.Combine(Path.GetTempPath(), Path.GetRandomFileName());
//🚀Step 1: Start Chrome
Console.WriteLine("🚀Starting a new Chrome instance...");
Directory.CreateDirectory(userDataDir);
var psi = new ProcessStartInfo
{
FileName = chromePath,
Arguments = string.Join(" ",
$"--remote-debugging-port={port}",
"--no-first-run",
$"--user-data-dir={userDataDir}",
"https://selectorshub.com/xpath-practice-page/")
};
var chromeProcess = Process.Start(psi);
if (chromeProcess == null)
{
Console.WriteLine("❌Failed to start Chrome.");
return;
}
Console.WriteLine("🚀Chrome started. Waiting for initialization...");
await Task.Delay(5000);
try
{
//✅Step 2: Get WebSocket Debugger URL
string? debuggerUrl = await GetPageWebSocketUrl();
Console.WriteLine(debuggerUrl);
if (string.IsNullOrEmpty(debuggerUrl))
{
Console.WriteLine("❌Failed to retrieve WebSocket Debugger URL.");
return;
}
// ⚙️Step 3: Connect to WebSocket
using var ws = new ClientWebSocket();
await ws.ConnectAsync(new Uri(debuggerUrl), CancellationToken.None);
//🚪Step 7: Close
Console.WriteLine("🚪Press Enter to close...");
Console.ReadLine();
await ws.CloseAsync(WebSocketCloseStatus.NormalClosure, "Close", CancellationToken.None);
}
catch (Exception ex)
{
Console.WriteLine($"❌ Error: {ex.Message}");
}
finally
{
if (!chromeProcess.HasExited)
{
chromeProcess.Kill();
}
}
}
Tasks
We need to enter text into an input element located inside a closed shadow root.
Visit this page, where I’ve identified an element within a closed shadow root.
Implementation
Since we can’t directly access the input element, we need to target its parent element using the #userPass
selector. In my previous article, I demonstrated how to retrieve an element's object ID—we’ll follow the same approach here.
// ✅Step 4: Get element
var hostId = await QuerySelector(ws, "#userPass", 1);
if (string.IsNullOrEmpty(hostId)) return;
Once we retrieve the ID, we need to scroll down to the element. Without this step, we can't focus on it, as it renders dynamically. Scrolling ensures the element is loaded into the viewport, making it accessible for interaction.
static async Task ScrollToElement(ClientWebSocket ws, string objectId, int id)
{
var command = new
{
id,
method = "Runtime.callFunctionOn",
@params = new
{
objectId,
functionDeclaration = "function() { this.scrollIntoView(true); }",
returnByValue = false
}
};
var response = await SendAndReceive(ws, command);
Console.WriteLine($"Click Response: {response}");
}
// ✅Step 5: Scroll
await ScrollToElement(ws, hostId, 2);
Next, to type text, we need to focus on the element. JavaScript provides a method for focusing elements, and CDP offers a specialized method called DOM.focus. However, there’s a challenge—the target element is a <div>
, which isn’t inherently focusable. Since it has no visible representation, the browser won’t allow focus on it.
To work around this, use the following code:
// ✅Step 6: Focus and type text
await PressArrowUp(ws, 3);
await PressTab(ws, 3);
await TypeText(ws, "helloworld", 4);
This approach comes with a few pitfalls. You might be wondering why I included the PressArrowUp
method. It’s a unique case—this component only loads when the user physically scrolls down. By simulating user interaction, this method helps trigger the loading process.
You can use different keys or methods to achieve the same effect, but I chose the Arrow Up key for this scenario.
static async Task PressArrowUp(ClientWebSocket ws, int id)
{
var keyDownCommand = new
{
id,
method = "Input.dispatchKeyEvent",
@params = new
{
type = "keyDown",
key = "ArrowUp",
code = "ArrowUp",
keyCode = 38
}
};
await SendAndReceive(ws, keyDownCommand);
var keyUpCommand = new
{
id = id + 1,
method = "Input.dispatchKeyEvent",
@params = new
{
type = "keyUp",
key = "ArrowUp",
code = "ArrowUp",
keyCode = 38
}
};
await SendAndReceive(ws, keyUpCommand);
}
After this step, you can press the TAB key to move the focus to the input element.
static async Task PressTab(ClientWebSocket ws, int id)
{
var command = new
{
id,
method = "Input.dispatchKeyEvent",
@params = new
{
type = "keyDown",
key = "Tab",
code = "Tab",
windowsVirtualKeyCode = 9,
nativeVirtualKeyCode = 9
}
};
await SendAndReceive(ws, command);
command = new
{
id = id + 1,
method = "Input.dispatchKeyEvent",
@params = new
{
type = "keyUp",
key = "Tab",
code = "Tab",
windowsVirtualKeyCode = 9,
nativeVirtualKeyCode = 9
}
};
await SendAndReceive(ws, command);
}
Finally, type the text into the input field.
static async Task TypeText(ClientWebSocket ws, string text, int startId)
{
int id = startId;
foreach (char c in text)
{
var command = new
{
id = id++,
method = "Input.dispatchKeyEvent",
@params = new
{
type = "keyDown",
text = c.ToString(),
unmodifiedText = c.ToString(),
key = c.ToString(),
code = c.ToString(),
keyCode = (int)c
}
};
await SendAndReceive(ws, command);
command = new
{
id = id++,
method = "Input.dispatchKeyEvent",
@params = new
{
type = "keyUp",
text = c.ToString(),
unmodifiedText = c.ToString(),
key = c.ToString(),
code = c.ToString(),
keyCode = (int)c
}
};
await SendAndReceive(ws, command);
}
}
Let's test it out. As you can see, the password input field has been successfully filled.
Bonus
When scraping or automating actions, using proxy servers is often essential. Typically, you might install a system-wide proxy application that affects your entire PC. However, I’ll show you how to configure a proxy locally, specifically for your project.
One way to achieve this is by creating a custom Chrome extension. This requires just a few lines of code along with a manifest file.
The manifest is a JSON file that defines settings and permissions for the extension.
{
"manifest_version": 3,
"name": "Proxy Auth Extension",
"version": "1.0",
"permissions": [
"proxy",
"storage",
"webRequest",
"webRequestAuthProvider",
"declarativeNetRequestWithHostAccess"
],
"host_permissions": [
"*://*/*"
],
"background": {
"service_worker": "background.js"
}
}
The second file allows you to fill in the credentials automatically.
chrome.runtime.onInstalled.addEventListener(() => {
console.log("Proxy Auth Extension installed.");
});
chrome.webRequest.onAuthRequired.addEventListener(
(details) => {
console.log("Auth request detected for:", details.url);
return {
authCredentials: {
username: "your_proxy_username",
password: "your_proxy_password",
}
};
},
{ urls: ["<all_urls>"] },
["blocking"]
);
The final step is to add the launch parameters.
var psi = new ProcessStartInfo
{
FileName = chromePath,
Arguments = string.Join(" ",
$"--remote-debugging-port={port}",
"--ignore-certificate-errors",
"--load-extension=/Users/serhiikorol/RiderProjects/Scarping/Scarping/Extensions/proxy-auth-extension",
"--proxy-server=http://brd.superproxy.io:33335",
"--no-first-run",
$"--user-data-dir={userDataDir}",
"https://selectorshub.com/xpath-practice-page/")
};
If you’ve entered the correct credentials, the target page will load successfully.
Final words
I’ve shown you an alternative approach to managing your browser. While it may seem more complex compared to Selenium or Puppeteer-Sharp, using the Chrome DevTools Protocol (CDP) directly provides greater control and flexibility.
NuGet packages can sometimes contain errors or compatibility issues with your current Chrome version. By using CDP directly, you can avoid these problems and ensure a more stable automation process.
I hope you found this article helpful. See you next time!
Source code here.