
Bulk Webpage Cacher
Github
Languages & Tools: Java
2 Programs, The first visits a list of websites from a text document automatically. The second creates html files from php based websites such as wordpress, which in turn creates cached pages.
This is mainly used to speed up caching webpages on large websites.
Recently used to create cached pages for 1.4 million product pages on http://electronikz.com.
Sample Code
count = thread -1;
while (count < urls.length) {
try {
document = Jsoup.connect(urls[count]).timeout(60000).get(); //get websitelink
completed ++;
System.out.println(count + " : " + "cached : " + urls[count] + " (" + completed + ")");
writer.append("success : " + urls[count] + "n");
count+=threadAmount;
attempts = 0;
}catch(Exception e){
System.out.println(count + " : " + "failed : " + urls[count] + " : Failed to cache post...");
writer.append("failed : " + urls[count] + "n");
count+=threadAmount;
}
}
System.out.println("Thread " + thread + " completed...");