The Problem

I can’t seem to read from a .txt file when I use read.csv inside a SQL 2016 stored procedure using R if that .txt file is on a network shared directory.

The Setup

I am running a personal domain in my home lab. The domain controller is running on an old pc I’ve repurposed to run Windows 2012 R2. On that server I run virtual machines using Hyper-V, one of which is a domain server with SQL 2016 Developer Edition installed. My desktop pc and laptop run Windows 10 and are members of my home domain. Both also have SQL 2016 Developer as well as R 3.3.1, RStudio 1.0.44 and R Tools for Visual Studio 0.4. In all cases SQL is configured to run external scripts.

On my domain controller I’m running a weather station program that writes to a comma delimited log file. There’s a separate file for each month. Each file has a row added every 10 minutes, and each row has 27 columns . There’s a separate text file that contains header information for the logs delimited by the pipe (|) character.

The Scenario

What I’m trying to do should be pretty easy. I want to use R code inside a stored procedure to read all the log files created by my weather program and store the results in a database. I also want to read the current monthly file on a regular basis, at least once a day. Once the data is in the database I’ll create some mobile reports with data and charts I can read on my phone. I’ll also be able to use my own data to play with local weather predicting. I thought it would make a pretty cool demo for my R sessions.

However when I run my stored procedure to read the logs I receive an error that there is no such file if I map the share as a drive, or I’m using an invalid parameter if I try to access the share directly.

What is even more confusing is that the same R code works if I run it from any other R interface; R console, Microsoft R Open console, RStudio, or RTVS.Since I can read the files in other ways from my client I’m ruling out permission or firewall issues.

The Code

The following code examples work in RStudio, R console, MS R Open console, and RTVS when run on the client. This connects to the share…

file_name <- paste(as.character(Sys.Date(), format = "%b%y"),"log.txt", sep="")
weather <- read.csv(file.path("//PERTELLVM01/Cumulus/Data", file_name),stringsAsFactors = FALSE, header=FALSE)
weathernames <- read.csv(file.path("//PERTELLVM01/Cumulus", "monthlyfileheader.txt"),sep = "|", header = TRUE)
names(weather) <- names(weathernames)
weather[,1] <- as.POSIXct(weather[,1], "%d/%m/%y", tz = "")
head(weather[, c(1:6)])

…and this connects when the share is mapped…

file_name <- paste(as.character(Sys.Date(), format = "%b%y"),"log.txt", sep="")
weather <- read.csv(file.path("w:/Data", file_name),
stringsAsFactors = FALSE, header=FALSE)
weathernames <- read.csv(file.path("w:", "monthlyfileheader.txt"),
sep = "|", header = TRUE)
names(weather) <- names(weathernames)
weather[,1] <- as.POSIXct(weather[,1], "%d/%m/%y", tz = "")
head(weather[, c(1:6)])

…and both return the correct data.

  Date..dd.mm.yy.  Time Temperature Humidity Dew.point Wind.speed
1      2016-11-01 00:00        59.7       67      48.8        9.2
2      2016-11-01 00:10        59.7       67      48.8        4.5
3      2016-11-01 00:20        59.9       68      49.3        5.4
4      2016-11-01 00:30        60.1       68      49.5        1.6
5      2016-11-01 00:40        60.3       69      50.1        3.8
6      2016-11-01 00:50        60.6       69      50.4        5.4

Now here’s the SQL statement running the R code when the share is mapped.

DECLARE @RScript NVARCHAR(MAX) = N'
file_name <- paste(as.character(Sys.Date(), format = "%b%y"),"log.txt", sep="")
weather <- read.csv(file.path("w:/Data", file_name),
stringsAsFactors = FALSE, header=FALSE)
weathernames <- read.csv(file.path("w:", "monthlyfileheader.txt"),
sep = "|", header = TRUE)
names(weather) <- names(weathernames)
weather[,1] <- as.POSIXct(weather[,1], "%d/%m/%y", tz = "")
OutputDataSet <- head(weather[, c(1:6)])'

EXECUTE sp_execute_external_script
@language = N'R',
@script = @RScript
WITH RESULT SETS (([Date] DATE, [Time] TIME(0), Temperature NUMERIC(4, 1), Humidity INT,
DewPoint NUMERIC(5, 1), WindSpeed NUMERIC(3, 1)));

Image may be NSFW.
Clik here to view.

Slightly different message when trying to access the share directly.

Image may be NSFW.
Clik here to view.

The Wrap Up

Now there are other ways I can import the data; copying the files I need to my client before running my R stored procedures or using SSIS packages to do all the importing just to name two. But I should be able to access the file directly with my procedure.

I’m curious if anyone else has run into this issue, and how they’ve solved it. Leave a comment if you can offer any insights. I’ve copied some sample files which you can access at https://github.com/jayape/Sample-Data.