The dark web has become the part of the internet where criminals prefer to conduct their business, from buying and selling stolen merchandise, obtaining resources, such as toolkits for ransomware attacks, and planning criminal acts. To put the dark web in perspective, it is a minor part of the deep web, which makes up between 96 to 99 percent of the entire internet. It is an important part of the shadow economy mimicking the legitimate economy. For example, Empire, a dark web marketplace listing over 6,000 products for sale from controlled substances to weapons and stolen credit card numbers. Buyers would order and pay in cryptocurrency and would then be contacted by a middleman for the shipment of the goods. All in all, 60 of the biggest websites on the dark web are around 750TB in size, which would take a lot of time to scan manually.

Since the dark web is not easily accessible and also risky, law enforcement, intelligence agencies, and private companies turn to dark web scanning solutions. Investigators use open-source dark web scanning software to keep track of threat actors, their networks, and their activities. Companies like to use a dark web scanning service to search the dark web for e.g., stolen usernames, passwords, Social Security numbers, and credit card numbers relating to their company that threat actors have put up for sale on the dark web.

Dark web scanning is part of cyber threat monitoring and intelligence, which is all the information an organization uses to understand the threats that have impacted their organization or might target them at any point in the future. To scan the dark web, which is an ongoing activity, organizations and agencies use online and automated tools such as monitoring server software for easy network monitoring. An automated, open-source dark web scanning service allows investigators in their efforts to prevent and identify potential threats aimed at e.g., infrastructure operators, intelligence agencies, government bodies, research centers, healthcare institutions, research institutes, financial and educational institutions, etc.

Such a dark web scanning solution can monitor thousands of chatrooms, social networks, and other potential online forums on the dark web in just a fraction of the time that it would take human investigators and analysts. Depending on the chosen solutions, the user can instruct the tool to alert the organization when flagged information, such as batches of credit card numbers or keywords and phrases used on dark web crime channels, appears to take preventive or reactive action.

Since threat actors are crafty, constantly looking for new dark web channels and technologies to minimize their chance of being caught during crime investigations, investigators and analysts need a sophisticated dark web scanning tool. Such an AI cybersecurity tool performs dark web monitoring services, scans dynamically and in real-time for sensitive information, keywords, and phrases entered into the search parameters by the users. Using smart algorithms such as natural language processing (NLP) and machine learning (ML), the monitoring solution will detect and collect the relevant big data from the surface, deep, and dark web for actionable follow-up. Moreover, the solution will collect this information continuously and constantly update its databases with real-time data. In other words, such a dark web scanning service will observe dynamic sites, including websites, online marketplaces, forums, and chatrooms, on the dark web where stolen information is bought and sold, or cyber-attacks are planned.

To recap, remaining aware and informed of potential cyber threats is crucial for both private organizations and intelligence agencies. It is, therefore, crucial to ensure that the AI monitoring solution used, such as the dark web scanning tool of Cobwebs, can perform dark web scans automatically, and can continually update its databases with new information gathered from around all layers of the web. It must also be scalable to adapt to an organization’s growing needs.