Go Lang

How to scrape page source with Go and chromedp

How to scrape page source with Go and chromedp

It’s clear what we are trying to achieve, so let’s consider the requirements. Firstly, we need a tool to render web pages since JavaScript is commonly used nowadays. Secondly, we require an API to communicate with the headless browser. Lastly, saving the result can be challenging as browsers are designed to interact with rendered results rather than directly with the source code.

Headless browser

So we are looking for a headless browser. We are going to use Chrome’s headless-shell because it’s easy to use, and it’s based on Chromium . The most significant advantage is docker image, which we can efficiently run on our local machine or anywhere in the cloud.

Golang and DynamoDB

Golang and DynamoDB

How to put data into dynamodb

I have been avoiding Amazon DynamoDB for a long time. Each time I tried to use it, either with PHP or GoLang always gave up very quickly. A while ago, I built up a solution storing a lot of JSON files on S3 buckets, and the costs of this solution (number of GET requests) started to escalate very quickly. I decided to try to migrate data to DynamoDB, and this time I was successful. I will try to explain how I did it and what I learned along the way.

How to create AWS Signature on GET POST request

How to create AWS Signature on GET POST request

Signing request with AWS Signature

Postman, it’s a great tool for debuging, testing and other activities performed by us - developers. It has a build in option to sign requestes with AWS Signature . We do have AWS SDK for Go v2 , but I couldn’t find any good examples except a test file . The other useful file will be const.go where we can find a couple of interesting variables.

How to implement timeout in go routines

How to implement timeout in go routines

Go lang simple workers with timeout using context

I wanted to implement a timeout in go routines. There are a couple o ways doing that. We could use context.WithTimeout() or we go with classical sync.WaitGroup group with a trick.

Let’s start with the first one.