Trying to get back on the horse

I want to incorporate this additional preferential parking data:

https://data.smgov.net/browse?category=Transportation

Questions:

• Is some or all of this data already in the existing dataset?

Approaching the deadline

Only small progress was made since my last post. A major factor was moving, which only took about a week but killed a lot of momentum. Fortunately I am settled in and back on track.

I had mentioned pre populating the Core Data storage in my app to avoid loading it on the client side, which was causing minutes long initial startup times. After much Googling I discovered such a technique was not supported. It seemed the only way to accomplish this was to write a separate app, load to the Core Data storage there, then copy over that data blob into my real app. Generally I prefer not to do things in a hack-y way, especially since future iOS, Xcode, or Swift updates could break something.

After struggling to find another solution, a random Stack Overflow comment recommended simply using a SQLite database. There was no good reason for me to use Core Data; I only went that route because I thought it was the normal storage method for iOS apps. I don't think that's the case, Core Data seems more about user specific storage (settings, etc). Once I settled on the database route it was smooth sailing.

Some of the implementation details had to change. The main difference is I'm no longer loading all the data. Instead I quickly load only the locations' latitude and longitude upfront. When determining the annotations' colors (allowed or not), two database queries give me exactly what I need.

With this change my app loads very quickly after install, and I've noticed no slowdowns when the annotations update every minute.

The issue of performance from too many annotations still exists however. I'm still only displaying 1000 points, and performance is already not great. I'm perpetually frustrated by this. How is it that Safari on my iPhone can use the web (javascript) version of my app and it works fairly well, yet this native app that is showing much fewer points runs so poorly? Makes me question my decision to go with Apple Maps over Google Maps.

Too many annotations

As with all my previous parking app implementations, the problem of "too many annotations" is a tough nut to crack once again. It's been a while since my last post, so let me describe out some of the progress I've made.

Swift doesn't have a natively easy way to work with JSON. I found an open source package online called SwiftyJSON that did a lot of the leg work. With it I was able to quickly parse my JSON file.

Loading the JSON takes a long time, on the order of minutes. It made sense to save this data using Core Data. The first time the app runs it spends a long time parsing the JSON, but subsequent app loads are much quicker. There must be a way to not have to do any JSON parsing in my app, and instead do it offline and save that data directly into my app. Worth exploring.

This brought us to the current problem. Displaying the approximately 10,000 parking signs as annotations causes the app to perform extremely poorly. I would classify it as unusable. My first instinct was to do some sort of clustering. Unfortunately Apple provides no implementation of such a feature, so I tried out several open source packages before settling on kingpin. It works okay, but it's implementation makes it hard to customize the annotations. And frankly it actually isn't very fast anyway.

In my search for inspiration I happened upon an app called Plane Finder. This app displays commercial flights in semi real time, displaying a plane icon at the purported location of the actual plane. On screen at any time are upwards of 5,000 plane icons, which I assume are annotations with custom art. Although it never displayed as many points as I did at once, the app is clearly doing something interesting that causes it to be very performant. I emailed the company asking if they'd be willing to share their technique. To my surprise they actually replied, but declined to share anything.

I considered going an underground route and reverse engineering their app, in hopes of figuring out how they're accomplishing their impressive annotation performance. The worst part about going that route is the need to jailbreak my device, which I really don't want to do. There's also no guarantee I'd actually learn anything valuable, so it's a gamble on my time. And of course there's the whole legal and ethical boundaries I'd be tiptoeing.

In the meantime I can work on other aspects of the app.

• App icon / logo. 
• Improved splash screen
• UI elements
• pan to user's current location
• pan to Santa Monica
• Preprocessing JSON data and saving to Core Data

Making my iOS app

My project was selected as one of the finalists in the Hack The Beach competition. As such I will continue working on it until the winner is chosen. The next logical step was to create an actual mobile app. I chose iOS because it's my preferred platform.

I initially dabbled in embedding a Google Maps view into the app. But after playing around, it just seemed a lot easier and more straightforward to use Apple Maps.

Through a mix of Youtube videos, Lynda.com tutorials and various online posts, I've gotten a simple app running. It displays a map, finds the user's location (after asking for permission), and plots a single parking point on the map.

Next, I want to plot all the points. There's probably a way to read a CSV or JSON file into the app.

Back on the horse

Since submitting my project to Challenge LA, I had completely stopped working on it. Barely gave it a thought actually. I figured I'd start looking for a job while waiting to hear back. The announcement of the winner(s) was to be on October 10th, which was this past Saturday. Did I win? Well, I don't know. Through a series of questionable circumstances the event was cancelled, with no future date given for any further developments. Government programs, am I right?

Luckily Santa Monica came through, in a way. They've created their own civic tech competition dubbed Hack the Beach. Very similar setup to Challenge LA, with the areas of focus being "community, mobility, and civic engagement".

I went to both events last week. I met a number of individuals from the City, including Francie and Kegan. They were helpful in pointing me to some existing datasets regarding street parking regulations. One dataset in particular looks very promising. I'll take a deeper look later.

Now the plan is to restart work on my project. I plan to do a 50/50 split between working on it while continuing interview prep. Deadline for the application is October 30th.

LA Street Park

I found a decent URL that I plan to stick with: lastreetpark.com !

I thought I needed to have a whole separate server. I went through some of the motions of cloning my existing AWS virtual machine. Turns out I didn't need to. I can set some configurations in the Apache server, and based on what hostname is pointing to my IP, I can serve files from my desired location. (Once configurations are changed, I needed to restart the server.)

For reference:

• Apache configurations live in /etc/apache2 , at least for my setup.
• I added a new VirtualHost tag in the ./sites-enabled/000-.... file. 

My configurations in Hover were apparently incorrect. URLs of the form like lastreetpark.com pointed to my server fine, but www.lastreetpark.com didn't. Instead it would redirect to a default Hover page about not having redirects enabled or something.

The fix was to modify both the "A" values in the DNS settings. There are two: the "@", and the "*". I had only set the "@" setting to my IP, but also setting the "*" seems to allow all URLs of the form *.lastreetpark.com to work. I needed to have a chat session with Hover to learn about this.

Fusion Table frustrations

As previously stated, I created a new column that encompasses all "no parking" and "other" type signs. I use this column in the filtering. Again, this is because Fusion Tables doesn't support OR operators. I didn't explore the possibility of using IN.

Now that I'm able to select between showing different types of signs, I wanted to represent the points as different colors. Red for no parking, green for other (I hesitate to call it "parking allowed" because there are some signs like "Taxi only" and what not, that isn't categorized properly yet.). I looked at many examples of other people's code, but couldn't figure out how to do it. I found that Derek's tutorial did talk about it. Essentially there's an interface within Fusion Tables that allows me to create "buckets", which are different values of a column. I can assign different colors to different buckets. Perfect! or so I thought. For some reason, I'm only able to choose between two buckets, "x" and "id". How utterly useless! I can't find any logical reason why that might be. I saw in the "select columns" chooser that "x" and "id" were the 10th and 11th columns respectively. I tried moving a different column in there, but no luck.

Another thing. Whenever I make a change to the data, whether its reordering the columns or setting colors for different buckets, there seems to be a delay before the data becomes usable. For example, if I load my map immediately after making a change, instead of my points I might see error messages across the map about how the data isn't available yet.

You can praise Google for a lot of things, but you can criticize a lot as well. Maybe I should look into CartoDB, just pay for it and be done with it.

--
The question is: now what? I don't know what features I'd want to add in the short time between now and the September 20th deadline.

Maybe an app to take photos and record the GPS location? Here's a quick and dirty pipeline idea:

photo of sign and GPS location -> OCR text from picture -> store in CSV -> run in normal pipeline

Viewable prototype

I finally feel comfortable showing my app. It's here: moretouching.com/parking

Currently the radio buttons control the day and time. Loading the page sets the correct values by the current time. Wanted to add new buttons to distinguish between no parking versus parking allowed. I tried doing this by the SQL query filtering, with something like this:

other stuff.. AND (no_parking='true' OR no_standing='true' OR no_stopping='true').

It wouldn't work. I spent about a day trying to figure out what was wrong. Turns out, Google Fusion Tables don't support the OR operator. That would've been useful information, anywhere in their documentation. I had to find out by looking at some Stack Overflow posts. (maybe there's a way to work around it using IN?)

As a work around I'm adding a new field called noParkingAll. Kind of clunky, but it's late and I'm distracted. Also makes me regret the no_parking style I went with earlier.

On a different note, I found some examples of using different colored dots for different types of points. Certainly something I'll want to do, with green and red dots. Maybe some orange and yellow in the future.

Data precision vastly improved

Since my last post I found many instances of  incorrect data. I won't go into too much detail, but here are the highlights:

1. Mouse over fusion tips seemed to be showing or causing incorrect data points to be displayed, disagreeing with the radio buttons. I turned them off for now, and instead rely on the pop up bubbles for information.
1. Turns out you can select the ordering of the columns, which effectively determines what comes up in the pop up bubble. There must be a way to select which fields to show here, but for now this works fine.
2. Some data points were showing incorrect data. It turned out to be an issue with how I was calling the parser. I didn't realize some parking descriptions had multiple sentences. The parser was treating the subsequent sentences of a description as the next description, so the data became misaligned with the descriptions after this point. Fixed by adding a new annotator option when calling the parser.
1. I initially thought it was related to times that were of the "11pm to 3am' flavor. It wasn't, but my code wasn't handling this case anyway, so I added it in.

At this point all the days and times seem to be displaying correctly corresponding to the descriptions.

What I'd like to do now is determine whether a sign says you can park somewhere, or it says you can't. I can display the differences via red and green dots. 

I'm going on my vacation soon. I want to keep working on this but I don't know if I'll be able to. Perhaps I shouldn't, even if I could. Debating taking my laptop with me on this trip.

Some hopeful developments

There have been a number of developments since I last wrote.

1. I successfully loaded all the points to the fusion table. It was much smaller as a CSV.
2. I modified the javascript so only points that fit the exact time criteria were shown, with appropriate radio buttons to control it.

For verification I wanted to look at the description of the points, but because I have so many fields, I couldn't see the description in the pop up bubble. In my attempts to figure out how to show only what I wanted in them, I decided to try deploying one of Derek Eder's existing projects called 2nd City Zoning. This process became a huge mess, but today I finally figured everything out. I'll try to outline everything I did here, in case anyone needs to replicate this.

• The instructions in his README seem simple, but to get it to work requires a lot of setup.
• A lot of it had to do with setting up the right Ruby things. This site provided most of what was needed. Some highlights:
• Install Homebrew, which in turn allows easy installation of other things.
• Install rbenv, which allows for custom ruby environments. Instead of using the built in OSX ruby, we can choose which ruby version to use, and easily switch between different ones.
• Ruby Gems is configured to install things to the local user directory.
• I created environment variables GEM_HOME and GEM_PATH, both set to a ".gem" directory in my home folder.
• Had a hell of a time figuring out why I was having problems. Turns out it was because I had quotes around the path names, like this: GEM_HOME="~/.gem".
• The correct way is shown below.
• GEM_HOME=~/.gem
• GEM_PATH=~/.gem
• I also installed Bundler, which takes care of installing all the dependencies.
• Seems to work with the "Gemfile" files.
• Still, I had trouble running jekyll, which complained about a dependency that I could see was installed.
• The answer was to execute the command using bundle, since I was managing the packages with it.
• The instructions also talks about adding a line to my bash profile, which perhaps removes the necessity of doing this. I don't know, haven't tried.

Now that I have it running I want to start playing with the javascript, so I can see how to modify what shows up in the pop up bubble. Then, I need to display the description.

I have a sneaking suspicion that the points aren't labeled properly because it doesn't pass the eye test. There seem to be too many no parking areas at night. Maybe this is actually correct, but I want to be sure.

---

I went to a hackathon on Sunday, and it was time very well spent. I got to meet LA's chief data person Abhi, who was hired by Mayor Garcetti to open up a treasure trove of LA's data. I asked him about the parking signs, and unfortunately he said that data doesn't exist in consolidated form. That's fine, I had a feeling. Still, I just know that it exists in some form, probably scattered, simply because of the existence of parking enforcement. How could they do their jobs without this information being available some how? We just need to get the data from them, and put it together.

There's a project competition due in September. I'm going to submit my project for it. The winners will get funded for 4 months, which sounds awesome. If I can't get the data I need by then, I might manually walk around a bunch of city blocks and record the data. Then show my project using this data, as a proof of concept. Hopefully it is compelling enough.

Fusion tables

I was having some luck getting fusion tables to work using Derek Ender's stuff. I added radio buttons to distinguish between days of the week. I wanted to add the same thing for times of day. A way to accomplish this is to have a fields corresponding to every time slot. Of course, this could greatly increase the size of the data. I wasn't sure if it would break fusion tables, nor did I know exactly how much, but I decided to go for it.

The resulting JSON turned out to be 1.23 GBs. Too big. The total size allowed is 1 GB, and each table can be up to 250 MB.

I was originally setting the fields as 'true' or 'false'. Changed it to 't' and 'f' in an effort to limit size.
---
Reduced it down to 1.08 GB... I mean, it seems small, but 0.15 GB is still 150 MB. Nothing to scoff at.

I could try reducing the field names too... But no matter what I do, it'll be tough to get the table size below 250 MB.

The original JSON with most fields cut was 142.7 MB, while the CSV was only 54.1 MB. That means the CSV was 38% of the original JSON size. We'd still be too big. I'd need to get the JSON down to 658 MB for the same math to work out for the 250 table size limit. Maybe try splitting the data by day?

Depression

http://www.wsj.com/articles/entrepreneur-finds-profitable-niche-parking-spaces-1414965251

So here's the guy who made Smooth Parking.

I've looked at the page's source code, and I think I know how he did it. I'm impressed he did all that honestly.

I'm very upset. At this moment I'm trying to build something that already exists, and is very good. Granted, my ultimate goal is to bring this to Los Angeles. However, I felt like I needed this NYC version to have some success to convince the city of LA to help me with this endeavor. Now, I feel that I have no shot at it. How can I compete with an already existing, well functioning, multi platform (both Android and iOS) version of something I haven't even made yet?

I feel like I've thrown away the past six months on this endeavor.

Some of the smallest silver linings:

The app doesn't seem popular. Why? I'd use it.
The aforementioned article was written in Nov 2014. He registered his domain in 2011. According to his LinkedIn, he started the company in mid 2012. It tells me that its taken years to get this thing off the ground.

---
I went for a walk. Had a lot of thoughts in my head. Also took a bit of time off to play some Hearthstone. I feel much better. I think I can make a pretty good web interface. Let's start there.

Prior art discouragement

I started finding existing products that do a lot of, or even do entirely what I want to do.

1. Street Parker has a nice interface. It doesn't show exactly what I want, but it has all the tools to make it happen. It also uses some open source (?) software for the interface. I want to look into that.

2. Smooth Parking is basically what I want, and it already exists! Dammit. They already have a nice web interface and mobile apps. 

3. An app that used to exist called PrimoSpot seems like it did a decent job at presenting parking info. I think I had read about it but for some reason the app is gone.

Where do I go from here? Do I continue this journey? To get to even Smooth Parking's level of features and polish will take me a very long time. I'm very discouraged.

I did notice that Smooth Parking's web interface treats times like I do - in 15 minute increments. I guess I wasn't completely crazy when I picked that. And the web interface is actually fairly slow when zoomed out. Plus, the mobile app doesn't show data until you're zoomed in up to a certain level, presumably because of performance.

I won't be discouraged. I think there is still hope.

Fixing errors

My logic for comparing start times and end times was funky. I cleaned it up to be much simpler, and now the times seem to match up correctly with the file names.

I noticed that "except Sunday"s were not working. I'd look at a map of Sunday, and see a bunch of parking signs that were supposed to invalid on that day. I looked at some code but it isn't obvious what the problem is.

Starting writing unit tests to address both these problems. They're great - they provide a nice amount of reassurance that the code works. They're also great for debugging, especially when tweaking regular expressions to get them just right.

Something seems up with getting the "except" regex. Seems like it should be obvious, since the simplest case of "except sunday" doesn't pass. Could be in the actual function, could be in the regex, could be a mistake in calling the wrong thing somewhere (wires crossed).

UPDATE

Figured out the "except" regex error. Now I'm re running the pipeline, and starting all the way back at "skipParse". I need all those intermediary steps which take care of getting the valid days.

Performance improvements

I realized that it was the creating of the index lists that was taking the longest time. I broke that portion out into a separate function, so I can bypass it and use existing index lists when creating new HTMLs. That way I can iterate quickly on creating modified HTMLs, like if I wanted to add more info to the marker mouse over (I've currently added the description).

I'm running into an old issue though - some files have too many markers, which seems to break Google Maps. HTML files around 2MB have just over 10,000 markers, and these don't open. The biggest files have 5.1MB, which have over 26,000 markers.

Not sure what to do about this. I can turn on the 10K limiter again, but then my results aren't "truth".

On the bright side the results do look good. At a cursory glance at some of the marker's descriptions, they seem to match up with the right time slot... maybe? Look at monday time slot 33, then look at the markers. Shouldn't time slot 33 be like 8:15 am? Looks like the markers are for like 7-730am.

Looking at some more data, some of it seems clearly wrong. Gotta revisit how I'm picking the indexes.

UPDATE

Figured out the issue with the "except" regex. Now re running, starting all the way back from "skipParse". Let's check it out in the morning.

Better display method

After reading some articles about using Google Maps I found a better solution to the current "too many markers" problem, called MarkerClusterer. It groups markers into clusters, and only displays cluster centers when zoomed out to a certain point. I have to download the given javascript source, then point to it from my HTML, then it just works. Performance is vastly improve, and it looks a lot better.

I still want to add more info to the markers.

An issue I'm having is the run time of creating all 672 HTML files. I timed it today, it takes almost 6 hours to run. I think I need to come up with a better way to implement and test repeatedly. I can cut down the number of points, or just try to improve the code. Right now, it scans through the entire JSON for every time slot, which comes out to something like 300,000 x 672, which is too much. The better approach would be to scan the JSON once, and find the appropriate index file(s) to write into. This adds complexity. I'll have to either keep all those files open and write to them correctly, which seems hard to organize. Or I could keep the index list's for each file, writing to them as I go, then write every file all at the very end. Or I could try to multithread the code and find a super powerful machine to run on. Finally, I recall Tomer talking about something called "pypy", which apparently runs python code but much faster.

Immediate next steps I want to tackle:

1. Display data through the markers, either mouse over text or click marker and bubble pops up.
2. Put the maps into a smaller "container" or "window", instead of taking up the whole web page. That way I can start adding buttons and sliders and what not to change between time slots.
3. Improve speed of creating HTMLs. Or, use smaller data set.

Time slots separated

I've created the unique HTMLs for every time slot, all 672 of them. The processing takes many hours to run.

I should've done this before, but I want to add information to the markers that are displayed. It'll make it much easier to verify correctness.

Also, now might be time to reinvestigate another displaying mechanism. Markers are not well suited to display this much information.

Date-Time comparisons

I started the process of creating separate HTMLs for every time slot. My previous calculations were wrong by the way. Not only are there 4 time slots per hour, 24 hours in a day, there are 7 days in a week. Thats 4 x 24 x 7 = 672! Whatever, no biggie.

One challenge I immediately ran into was comparing times. The parser spits out times in a text, possibly in the TIMEX3 format according to the documentation. I couldn't find any intelligible way to use this, but did find a useful python package called date-util. It has a parser itself that takes most time formats like "11:35am" or "14:00" and converts it into a usable format. I used this to make the necessary calculations between times.

I want to refactor the way I'm using file names and directories. I've been using filenames with hardcoded directories prepended, but that doesn't seem like the right way to go. Maybe I'll tackle that tomorrow.

Learning from data

I implemented the features to find "anytime" tokens, as well as days of the week, and stored them into variables of the JSON. I wanted to handle the case of "except" seen before a day of the week, for example "... except Sunday". I noticed that almost all instances of this type were for exactly Sunday. I got the count of these words. Note, this is before normalizing abbreviations:

• sunday - 34,504
• sun - 4492
• trucks - 3931
• authorized - 2871
• school - 630
• su - 286
• commercial - 266
• vehicles - 164
• farmers - 129
• saturday - 85
• taxis - 64
• comm - 61
• miu - 29
• city - 26
• buses - 21
• pick - 20
• tlc - 18
• taxis/fhv's - 17
• loading - 16
• license - 6
• tour - 6
• m - 6
• horse - 5
• nycta - 3
• sunda - 3
• flea - 3
• greenmarket - 3
• friday - 2
• 8am-11am - 2
• n - 2
• state - 2
• mon - 2
• us - 1
• deliveries - 1
• sunady - 1

Clearly Sunday is the major day. Notice even the variations of Sunday; the multiple types of abbreviations, and the one misspelling. 

After this, I should be able to create the separate HTMLs for every time slot. Let's actually calculate how many there will be.

24 hours in a day

4 15-minute blocks per hour

24 * 4 = 96

That's not so bad.

Post project

I wrote my writeup, and have gotten at least verbal approval from all reviewers. I'm supposed to see Millstein tomorrow, and then I should be home free!

What comes next? I think adding a few additional elements will give us a presentable web interface product.

1. Account for parking signs that say "anytime" (without any other TIME elements?)
2. Figure out the days of the week that a sign is valid.
1. Range of days, or
2. Specific days
3. Create separate HTML files based on 15 minute increments
4. Create a basic web interface where the user gives the day of week and time
1. drop down menus for the days?
2. slider for the time?
3. radio buttons for hour, minute, and AM vs PM?
5. Display data as green and red lines, instead of markers.

Once I have these steps complete, I think I can have people start using the service.