![\"\"\/](\"https:\/\/cdn-images-1.medium.com\/max\/600\/1*CUzmB6bO9i9SB8lqtc9D8Q.png\")
You will notice pretty fast, that if you are not an early bird, all permits for your desired date are gone. Now you have three choices. One, resign and don\u2019t hike the mountain. Two, check the website by yourself every day, to see if new or returned permits are available. Three, get a bot or browser automation to work that checks the permits for you. My friend decided to ask me for the third option. And as I have some experience with RSelenium (just recently presented at EARLconf<\/a>), I wanted to try this approach.<\/p>\n\n\n\n First I wanted to run my bot <\/em>on the cloud. Moreover, I wanted to get a reproducible environment. So I decided to follow the vignette approach <\/a>of RSelenium. This means using a docker container for Selenium. So my first task was to spin up two docker containers. The first one should run Selenium, the second one should run R and python to access it.<\/p>\n\n\n\n Spinning up the Selenium container is simple:<\/p>\n\n\n\n I used a shared network between the docker containers called The second docker container must consist of three files.<\/p>\n\n\n\n The Dockerfile needs to look like this:<\/p>\n\n\n I used Now I would like to spin up the docker container. But my To use RSelenium you first need to connect to the Selenium server. It runs in the other docker container. To connect to it run:<\/p>\n\n\n\n The name The much harder part is to find the elements to click on. So first I noticed, that I need to click on the option \u201cAll Routes\u201d, which was the third one from the dropdown menu:<\/p>\n\n\n\n This option can be accessed by its As you can see This was the easiest part of browser automation steps. The much harder part is entering the number of hikers. The safest way to change them is not only to type into the text field but also to use javascript to change its value. The field To change the value I used the following code:<\/p>\n\n\n\n You can clearly see that I wanted one single permit for the mountain. The javascript piece ran on the webElement itself, which was stored in After these steps, the calendar with permits gets activated. I wanted to get a permit in October 2019. So I needed to click on \u201cNEXT\u201d until October shows up.<\/p>\n\n\n\n I build a loop to perform this task using the\u00a0 The element containing the month was had the tag Finding an available day as a human is simple. Just look at the calendar and search for blue boxes with an A inside:<\/p>\n\n\n\n For a computer, it is not that easy. In my case, I just had one question to answer. What is the first date in October 2019 to climb Mt. Whitney?<\/p>\n\n\n Thus I searched for any entry with the class\u00a0 Once the script was finished I wrapped up all files in my GitHub repository (zappingseb\/mtwhitney)<\/a>. They will help me by sending an email every 12 hours. From there I went back to my docker server and git cloned the repository. I could then spin up my docker container by running:<\/p>\n\n\n and test the script one time by connecting to the docker container:<\/p>\n\n\n and running the script with:<\/p>\n\n\n I got an email. After receiving the email I was sure the script will run and disconnected using Scripting this piece really got me an email with a free slot and a permit to climb Mt Whitney:<\/p>\n\n\n\n Browser automation can be helpful for use cases like this one. It helped me to get a permit. I did not overengineer it by scraping the website every few seconds or checking for specific dates. It was more of a fun project. You can think of a lot of ways to make it better. For example, it could send an email, if a date is available. But I wanted to get the log files every 12 hours, to see if something went wrong.<\/p>\n\n\n\n During the scraping, the website got updated once. So I received an error message from my scraper. I changed the script to the one presented in this blog post. The script may not work if you want to scrape Mt.Whitney pages tomorrow. Recreation.gov might have changed the website already.<\/p>\n\n\n\n I use browser tests to make my R-shiny apps safer. I work in a regulated environment and there these tests safe me a lot of time. This time I would have spent click-testing without such great tools as RSelenium <\/a>or shinytest<\/a>. Try it out and enjoy your browser doing the job for you.<\/p>\n","protected":false},"excerpt":{"rendered":" Mount Whitney is the tallest mountain in the contiguous United States and you need a permit to climb it. These permits are limited. But from time to time, somebody will […]<\/p>\n","protected":false},"author":1,"featured_media":4412,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"ngg_post_thumbnail":0,"footnotes":""},"categories":[1],"tags":[384,427,428,426,423,381],"_links":{"self":[{"href":"https:\/\/engel-wolf.com\/index.php?rest_route=\/wp\/v2\/posts\/4411"}],"collection":[{"href":"https:\/\/engel-wolf.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/engel-wolf.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/engel-wolf.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/engel-wolf.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=4411"}],"version-history":[{"count":1,"href":"https:\/\/engel-wolf.com\/index.php?rest_route=\/wp\/v2\/posts\/4411\/revisions"}],"predecessor-version":[{"id":4413,"href":"https:\/\/engel-wolf.com\/index.php?rest_route=\/wp\/v2\/posts\/4411\/revisions\/4413"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/engel-wolf.com\/index.php?rest_route=\/wp\/v2\/media\/4412"}],"wp:attachment":[{"href":"https:\/\/engel-wolf.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=4411"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/engel-wolf.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=4411"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/engel-wolf.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=4411"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}The knowledge you need to follow this tutorial<\/h3>\n\n\n\n
Getting started with RSelenium and docker<\/h3>\n\n\n\n
docker run -d -p 4445:4444 --name seleniumcontainer --net mynet selenium\/standalone-chrome<\/pre>\n\n\n\n
mynet<\/code> . This allows the two docker containers to find each other in the network, even by names.<\/p>\n\n\n\nrun_tests.R<\/code> to execute my RSelenium calls<\/li>sendmail.py<\/code> to send emails from python<\/li>DockerFile<\/code> to build a docker container<\/li><\/ol>\n\n\n\n\n# alpine-python-ecmwfapi\nFROM rocker\/tidyverse:3.6.0\nMAINTAINER zappingseb "sebastian@engel-wolf.com"\n\nRUN R -e "install.packages(c('RSelenium'), repos='https:\/\/cran.rstudio.com\/') "\n\nRUN apt-get update -qq \\\n && apt-get install -y \\\n python-pip \\\n vim\n\nRUN pip install pytest-shutil\nRUN pip install --upgrade numpy secure-smtplib email\n\nCOPY run_tests.R \/tmp\/run_tests.R\nCOPY sendmail.py \/tmp\/sendmail.py\n\nRUN apt-get update && apt-get -y install cron\nRUN echo "0 *\/12 * * * root Rscript \/tmp\/run_tests.R" >> \/etc\/crontab\nRUN service cron start\n<\/pre><\/div>\n\n\ntidyverse<\/code> docker container and installed the RSelenium package. Additionally, I installed python secure-smtplib<\/code> and email<\/code> . I also already added a cronjob to my docker container. This cronjob will run the web crawler every twelve hours by:<\/p>\n\n\n\nRUN apt-get update && apt-get -y install cron
RUN echo \"0 *\/12 * * * root Rscript \/tmp\/run_tests.R\" >>
\/etc\/crontab
RUN service cron start<\/pre>\n\n\n\nsendmail.py<\/code> and run_tests.R<\/code> files were missing. Let\u2019s create them<\/p>\n\n\n\nUsing RSelenium to crawl permits<\/h3>\n\n\n\n
remDr <- remoteDriver(remoteServerAddr = \"seleniumcontainer\", browserName = \"chrome\")<\/pre>\n\n\n\n
seleniumcontainer<\/code> will be automatically identified as long as the container runs inside mynet.<\/code> Two steps will lead to the Mt. Whitney permit website. Opening a browser and navigating to the website:<\/p>\n\n\n\nremDr$open()
remDr$navigate(\"https:\/\/www.recreation.gov\/permits\/233260\/\")<\/pre>\n\n\n\nWork with the permit form<\/h4>\n\n\n\n
![\"\"\/](\"https:\/\/cdn-images-1.medium.com\/max\/800\/1*2a0OGb9qickeQ_aXuyI7zg.png\")
id<\/code> . This id is division-selection<\/code> . By clicking on the element with the id<\/code> , the dropdown will open. After the dropdown is open, you need to click on the 3rd option<\/code> element available on the website. With these 4 lines of code you can realize it using RSelenium:<\/p>\n\n\n\nel_1 <- remDr$findElements(\"id\", \"division-selection\")<\/pre>\n\n\n\n
el_1[[1]]$clickElement()<\/pre>\n\n\n\n
el_2 <- remDr$findElements(\"css selector\", \"option\")<\/pre>\n\n\n\n
el_2[[3]]$clickElement()<\/pre>\n\n\n\n
findElements<\/code> returns a list of webElements <\/em>with the desired attributes. clickElement<\/code> is a method of such a webElement <\/em>and will basically click the element.<\/p>\n\n\n\nnumber-input-<\/code> will be used for this.<\/p>\n\n\n\n![\"\"\/](\"https:\/\/cdn-images-1.medium.com\/max\/800\/1*JoLQD_-t414hjRLC3IYj5g.png\")
el_3 <- remDr$findElements(\"id\", \"number-input-\")<\/pre>\n\n\n\n
# executing a javascript piece to update the field value
remDr$executeScript(\"arguments[0].setAttribute('value','1');\"), list(el_3[[1]]))<\/pre>\n\n\n\n# clearing the element and entering 1 participant
el_3[[1]]$clearElement()<\/pre>\n\n\n\nel_3[[1]]$sendKeysToElement(list(\"1\"))<\/pre>\n\n\n\n
el_3[[1]]<\/code> . For RSelenium I prefer finding elements with the remDr$findElements<\/code> method. Afterward, I take the first piece if I am sure that there is just a single element. The methods clearElement<\/code> and sendKeysToElement<\/code> remove old values and enter the value needed. The API of sendKeysToElement<\/code> is a bit weird, as it requires a list of keys, instead of a string. But once used, it is easy to keep your code.<\/p>\n\n\n\nInteract with the permit calendar<\/h4>\n\n\n\n
![\"\"\/](\"https:\/\/cdn-images-1.medium.com\/max\/800\/1*FQoSyX94WlxiGBZXuzrAiw.png\")
while<\/code>\u00a0command<\/p>\n\n\n\n# Get the initial month shown\nmonth_elem <- remDr$findElements("css selector", ".CalendarMonth_caption strong")\nmonth <- month_elem[[1]]$getElementText()\n# Loop to until the October calendar is shown\nwhile(!grepl("October", month)) {\n \n el_4 <- remDr$findElements("css selector", ".sarsa-day-picker-\n range-controller-month-navigation-button.right")\n el_4[[1]]$clickElement()\n Sys.sleep(1)\n month_elem <- remDr$findElements("css selector", \n ".CalendarMonth_caption")\n month <- month_elem[[2]]$getElementText()\n}\n<\/pre><\/div>\n\n\nclass=\"CalendarMonth_caption\"><strong>...<\/<\/code> . I accessed this with a CSS selector. Upon clicking the next button, which had a specific CSS class, a new calendar name shows up. It took me a while to find out that the old calendar month is not gone. Now the second element has to be checked for the name. So I overwrite the month<\/code> variable with the newly shown up heading of the calendar.<\/p>\n\n\n\nDerive the first available date from the calendar<\/h4>\n\n\n\n
![\"\"\/](\"https:\/\/cdn-images-1.medium.com\/max\/800\/1*WnPWGRtgU1uKGwJz1quywQ.png\")
\nday_elem <- remDr$findElements("css selector", ".rec-available-day")\nif (length(day_elem) < 1) {\n earliest_day <- "NODAY"\n} else {\n earliest_day <- strsplit(\n day_elem[[1]]$getElementText()[[1]],\n split = "\\n")[[1]][1]\n}\n<\/pre><\/div>\n\n\nrec-available-day<\/code>\u00a0. In case any entry was there, I got the text of the first one and took all characters before a line-break. This extracted the number of the date. Now, wrap this up and send an email with the date: <\/p>\n\n\n\nfileConn<-file("\/tmp\/output.txt")\nwriteLines(paste0("The earliest day for Mnt Whitney in ", month[[1]], " is: ", earliest_day, "th of October 2019.\\n\\n-------------------\\n"))\nclose(fileConn)\n# Write an email from output.txt with python\nsystem("python \/tmp\/sendmail.py")\n<\/pre><\/div>\n\n\nActivating the docker container<\/h3>\n\n\n\n
\ndocker build --net mynet --name mtwhitney -f Dockerfile .\n<\/pre><\/div>\n\n\n
\ndocker run -it mtwhitney \/bin\/bash\n<\/pre><\/div>\n\n\n
\nsudo Rscript \/tmp\/run_tests.R\n<\/pre><\/div>\n\n\n
Ctrl + p<\/code> and Ctrl + q<\/code> .<\/p>\n\n\n\nLearnings<\/h3>\n\n\n\n
![\"\"\/](\"https:\/\/cdn-images-1.medium.com\/max\/800\/1*EQw4FDV3nfqjYupvJaSArA.png\")