The immediate solution was make the application tolerant of orphaned rows in the child table and to drop the explicit foreign key constraint.

I didn’t really want to leave those orphaned rows hanging around in the child table, so I decided to implement an asynchronous process to delete the orphaned rows on a scheduled basis. Read on for a description of that process.

Using the sakila database as an example, imagine I drop the foreign key between file_category and category, like so:

1
2

alter table sakila.film_category 
  drop foreign key fk_film_category_category;

Without the foreign key in place, deletes on the category table lead to orphaned rows in film_category. For example, I will delete the “New” category:

1
2
3

mysql> delete from sakila.category
    -> where name = 'New';
Query OK, 1 row affected (0.01 sec)

There are several ways to count the orphaned rows. Here are two different naive implementations using OUTER JOIN or NOT EXISTS:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

mysql> select count(*)
    -> from sakila.film_category c 
    -> left outer join sakila.category p on p.category_id = c.category_id 
    -> where p.category_id is null;
+----------+
| count(*) |
+----------+
|       63 |
+----------+
1 row in set (0.00 sec)

mysql> select count(*)
    -> from sakila.film_category c 
    -> where not exists 
    -> (
    ->   select NULL from sakila.category p where p.category_id = c.category_id
    -> );

+----------+
| count(*) |
+----------+
|       63 |
+----------+
1 row in set (0.01 sec)

I can also delete the orphaned rows using the same query approaches. I’ll roll the first delete back so I can demonstrate the second query in the same session:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

mysql> set autocommit = 0;
Query OK, 0 rows affected (0.00 sec)

mysql> delete c.*
    -> from sakila.film_category c 
    -> left outer join sakila.category p on p.category_id = c.category_id 
    -> where p.category_id is null;
Query OK, 63 rows affected (0.02 sec)

mysql> rollback;
Query OK, 0 rows affected (0.00 sec)

mysql> delete c.*
    -> from sakila.film_category c 
    -> where not exists (select NULL from sakila.category p where p.category_id = c.category_id);
Query OK, 63 rows affected (0.01 sec)

mysql> commit;
Query OK, 0 rows affected (0.00 sec)

This approach will be very slow for tables containing millions of rows, so in my real world case I didn’t use this approach. Instead I decided it would be a lot easier and faster to delete the orphaned rows if I knew who their parent was. To this end I created a new table to track the deleted rows, and populated it using a trigger. Continuing the example in the sakila database:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

CREATE TABLE category_deleted (
  category_id tinyint(3) unsigned NOT NULL,
  name varchar(25) NOT NULL,
  last_update timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  delete_time timestamp NOT NULL,
  PRIMARY KEY (category_id)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

DELIMITER $$

DROP TRIGGER IF EXISTS sakila.TR_A_DEL_CATEGORY $$

CREATE TRIGGER sakila.TR_A_DEL_CATEGORY AFTER DELETE ON sakila.category FOR EACH ROW BEGIN
  
  INSERT IGNORE INTO sakila.category_deleted (category_id, name, last_update, delete_time)
  VALUES (old.category_id, old.name, old.last_update,now());

END $$

DELIMITER ;

Now I can delete another category to test the trigger:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

mysql> -- delete a single category
mysql> delete from sakila.category
    ->     where name = 'Classics';
Query OK, 1 row affected (0.01 sec)

mysql> -- verify the trigger worked
mysql> select * from sakila.category_deleted;
+-------------+----------+---------------------+---------------------+
| category_id | name     | last_update         | delete_time         |
+-------------+----------+---------------------+---------------------+
|           4 | Classics | 2006-02-15 04:46:27 | 2013-05-21 18:21:53 |
+-------------+----------+---------------------+---------------------+
1 row in set (0.00 sec)

mysql> -- count the orphaned rows using the _deleted table
mysql> select count(*)
    -> from sakila.film_category c 
    ->     inner join sakila.category_deleted p on p.category_id = c.category_id;   
+----------+
| count(*) |
+----------+
|       57 |
+----------+
1 row in set (0.00 sec)

I also wanted to execute the deletes on the child table in chunks, so I implemented a stored procedure to delete the orphaned rows by iterating through the rows in the _deleted table, deleting the child rows, and then deleting from the _deleted table. If I had to implement it again, I would probably use common_schema to chunk the deletes so I wouldn’t need the stored procedure.

Here’s an implementation using common_schema:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

-- first delete the orphaned rows from the child table
set @script := "
  split(sakila.film_category: delete sakila.film_category.*
    from sakila.film_category 
    inner join sakila.category_deleted on sakila.category_deleted.category_id = sakila.film_category.category_id 
  ) 
  SELECT $split_total_rowcount AS 'rows deleted so far';
";
call common_schema.run(@script);

-- then delete the rows from the _deleted table (assuming they have no children)
delete p.*
from sakila.category_deleted p 
left outer join sakila.film_category c on c.category_id = p.category_id 
where c.category_id is null;

This interested us, and looking at the plethora of public data sets available on the web, we kept coming back to the sets that hit closest to home: San Francisco City data.

The San Francisco Open Data Portal provides a wide range of data sets containing information related to campaign finance contribution, public transportation (MUNI), and even listings of locations for movies shot in the city.

Being somewhat typical, modern city dwellers, one thing we love about San Francisco is the many purveyors of, and availability for, tasty food and beverage. Given this preoccupation with eats, drinks, etc., one of the most interesting data sets available to us was the Health Department’s repository of Restaurant Health Inspection Scores.

We set out to build a visualization of these scores by plotting this data in an application using Google Maps.

Red sections indicate lower (high risk) health scores. Blue sections represent higher (low risk) health scores.

Our hope was to provide a heat map of the city indicating areas that have more dubious or more positive health scores. Ultimately the visualization provides more of a heat map of restaurants inspected over a period of time. Looking at the results, we came to the conclusion that health score filtering options would be necessary to accomplish what we were after.

For a fully functional visualization of health scores on a map, visit sfscores.com.

Bon Appétit!

Team Asparagus (Steve Rowe, Saami Siddiqui, Omar Megdadi, Eugene Feingold, John Skinner)

When a user interacts with a Flite ad, we collect all actions made by the user within the ad. The common interactions we collect include clicks, scrolls, hovers, clickthroughs and many more. Each interaction contains a series of information, including the sequence number, the spot where the interaction happened (x,y position relative to the ad), component name, etc. With this information we can reconstruct the whole user session.

As we have this huge amount of interaction data, we thought of building a feature which shows the most common user path for a specific ad. This can help data analysts see how our current users are interacting with the ad, when they are leaving the ad, and find patterns to improve the interaction rate.

For any given ad:

1. Grab all interaction data, grouped by session.
2. For each session, remove interactions that are not important like hovers, rollovers, rollouts etc., and generate new sequence numbers.
3. Group all sessions by its sequence number and component name.

After that, we used the data to construct a weighted tree where the root node will always be the number of impressions served till now. The child nodes for the root will be the first interactions users have done on the ad. You can click on each node to find out what other interactions users performed after that interaction.

Session Ended events show how many users didn’t interact with the ad after the parent interaction.

To implement the UI, we used the flow plugin from an open source visualization tool - D3.flow.js. On each node click we make an ajax call with the id to get the child nodes and render them. The time consuming part was grabbing session data for an ad and constructing a data model out of the huge data set within 24 hours.

Cheers

Team Beets - Sam Chan, Toshi, Jiangyue, Chakri

The plan was to gather the smallest, but most relevant data set as quickly as possible, see what it told us, and show it off. Ike and Angel mined our own data in Hive for the top impressions, interactions, and browser by city and state. Meanwhile, Andy and Matt decided to see what kind census data we could get from census.gov regarding population, income, gender, and employment rate. Once we had all of that info, Ike was going to merge the data in MySQL, Angel would then analyze it for trends, and Andy would build a page to display the data.

Before we went out and started gathering data, we decided to form a hypothesis about correlations between census data and Flite usage data. The best idea we could come up with based on state-level data was that a higher unemployment rate may correlate with a higher time on unit, engagement rate, and/or interaction rate. Read on to find out if our hypothesis was valid.

During the data gathering processed we discovered a few interesting things and ran into a few hiccups. For example, the team discovered that West Virginia has the highest usage of Internet Explorer on Flite ads. The lowest: Utah. (Utah, you’re awesome. Thank you.)

We also discovered that the data on census.gov wasn’t always easy to procure the way we wanted it. One on hand making an API call to the 2010 census data was straightforward and fast. Matt was able to get a unique key to make the calls within minutes and was running our first queries in less than 15 minutes. On the other hand, the API calls to the American Community Survey (ACS) 5 Year Data Set was much more difficult and we never made a successful custom query. For the most part this was due to the lack of clear documentation on the census.gov site.

Since we needed some additional information and were unable to get it from ACS we went looking elsewhere. When in doubt, look it up on the “The Google,” right? Andy found a couple of pre-populated pages on bls.gov and census.gov for unemployment numbers and median state income. We weren’t able to easily export the data so Matt just hacked together his own CSV files and sent them to Angel.

We had a little more time and thought that gathering population information by city in the US would be fun to look at as well. Fortunately, Ike found this population data fairly quickly from biggestuscities.com which linked directly to the relevant data set on the census site. Unfortunately the city/town names in that data set did not match up well with our city-level Flite usage data, and we didn’t have enough time to clean the data set before demos.

So what did we find? First, although humorous, we tossed out the browser data because we didn’t have time to integrate it. Matt really dislikes IE so it was best not to go down that rat hole.

Looking at the maps comparing Flite usage data to population data, the first thing we calculated was daily Flite ad impressions per 10,000 residents. That map didn’t look very interesting because we didn’t see much variation between the states. When we looked more closely at the data set we found that Washington, DC (which we did not highlight on the map) had by far the highest per capita impression rate, about 300% of the second place state. Ike’s theory is that more people work in DC than actually live there. We found a blog post that supported our theory about DC’s high employment/residence ratio, higher than any other US city. And unlike other US cities where the effect is dilluted over an entire state’s population, Washington, DC is both a city and a state for our purposes.

Keeping it political, Ohio was a true battleground state in the 2012 election, but it’s the clear winner when it comes to Clickthrough Rate. We reached out to our in-office Ohio expert, John Skinner, but received no comment.

Engagement Rate and Interaction Rate by state also had some interesting variations, but we couldn’t come with any useful theories about the cause:

Finally, and the truly most interesting part of this project is that Angel discovered that that higher the income rate, the lower an ad’s interaction rate ended up being. We compared all of the different factors, and found two strong correlations.

The first was that interaction rate and median income were inversely proportional:

Secondly, following up on our initial hypothesis we found a positive correlation between engagement rate and unemployment rate:

This data supported our initial hypothesis to some extent. Of course there are many caveats working with state level data. For example, many ad campaigns are regional, so some of the data trends may be based more on the ads themselves rather than the people interacting with them. Also, the “unemployment rate” we used was not the true unemployment rate, but rather we divided the census figure for unemployed persons by the census population figure. By the time we realized that error it was too late to re-run the data with official unemployment rate figures. Nonetheless, it was nice to find some relationship within our data set!

Keeping this in mind, Hackday 2013 was kicked off with an Iron Chef theme, culinary-based team names (go Trout!!), and a Hacker draft.

The “Chairman” unveiled the special ingredient for this Hackday challenge: Big Data. Additionally, data resources were distributed to all teams and a firm deadline of 4PM was established.

Following a quick Trout brainstorm session, we were able to establish certain goals for our Hackday. Team goals were to track interactions as a means of gaining insight to platform usage as well as to utilize both Flickr’s large photo repository and public-use metadata.

With this challenge rooted in the use of data, the initial steps were deciphering where to source data from and how to mesh the information into a useful format.

Interaction tracking data was sourced within the platform, the random-sample data was aggregated interactions across both weekends and weekdays (20,000 sessions per day) for a month long interval. After statistical analysis and visual inspection the expected results were not realized. Initial expectations were that a large difference between weekday and weekend interactions should present itself within the data. Assumptions included: employees at work during the week would fuel interactions and that weekends would be spent enjoying leisure activities meaning more time away from the “point” of interaction. However, the data tells a different or conflicting story. Graphics show similar interaction data and usage habits across both weekends and weekdays with a horizontal shift (2 hours to the right). This horizontal shift could express the later start to people’s days during the weekend as well as their tendency to stay up later on weekend nights into the AM, fueling the higher weekend interactions post midnight when compared to weekdays.

Additionally, the confidence interval is much tighter for weekday interactions when compared to the wider CI’s associated with weekend interactions. This can be attributable to the predictability of one’s workday hours and interactions when compared to the less predictable and greater variance presented by less regimented weekend days.

UFO Data & Flickr Photos

Within the group there was an affinity to make the most of the UFO data available from the National UFO Reporting Center. The UFO data was refined and filtered, leaving the desired cell-based variables including:

• City/State
• color associated with UFO sighting
• gender of claimant
• shape of UFO
• use of term “alien” in the sighting description

Team focus shifted towards the use of this observational UFO data as a means of forming queries within the Flickr platform. The Flickr platform is rich with tags, locations, and personal descriptions that should match well with our workable dataset. Matching text data with image tag search results from Flickr should yield photos of these UFOs and in turn lead to visual verification of these sightings.

Initial obstacles included data formatting issues (text versus numerical). There were also problems with Flickr search results producing different responses based on manual versus automated queries. Strangely, manual searches yielded more results than automated, which essentially makes our service offered less effective and negates the applicable use of our creation. Limited auto query results were due to geographic tagging variation between Flickr (numerical) and UFO (text) data as well as the ability to include all search terms jointly using an “or” syntax.

Less than stellar query results continued, so as part of the iterative process we loosened our search parameters and tags. Exclusion of gender and color associated with UFO sightings seemed to increase query results and increase the effectiveness of our search. However, with this minimization of specified variables there is a decrease in test strength and results significance.

Following the discard of several search variables and adjustments of geographic tags into numerical data we were able to produce query results that exceeded the threshold we established as our benchmark. With the desired results in our query the JPEG photos were “in-filed” to JSON and uploaded to a user interface for the 4 PM demo.

At Flite, we have no idea what type of content is displayed alongside our ads, which run millions of impressions all over the web. The engagement and interaction rates are mostly based on the creative and messaging. But this may not always be the complete picture. The context of the real world in which the ad runs also may have a great amount of influence on these rates.

For our hackday project, we wanted to pull in data sets from many sources and measure the numbers against the engagement and interaction rates to see if there are any correlations within our Report Studio.

We started the process by looking for the apporiate data sets that would be apllicable to our project. Some of our challenges included the following:

• Free or public data is hard to find, especially daily or historical data
• Data in various formats made it hard to parse
• Google APIs are inaccessible, or not available/rate-limited

The stock index (S&P 500) ended up providing us with the cleanest data set. We added a UI in our Report Studio that would enable users to input a stock symbol and see whether we could correlate that data to our metrics on the ad impression.

In the Report Studio, we use a Javascript library called Highcharts to give a visual representation of the data to our customers. During the hackday process, we found out that we are on an older version of Highcharts, and upgrading was not straightforward. Part of the project was to be able to dynamically add in new series to the chart that has its own yAxis and scale. This feature was only available in the 3.0 version of highcharts and we had 2.4. After upgrading, there seemed to be some small quirks with compatibility with our code.

We also explored how we could correlate open data if we had more granular geolocation data available in the Studio. We collect geolocation data at the ad level and that can currently be found under the Measure tab in the Ad Studio; however, that data is not available in Report Studio (yet). If it were available, we could tell the user the geographic location where the most impressions were served and information about that location, like headlines, events or weather that happened there on that day.

Impression data could also be represented on a map using geolocation to see if local news events could possibly have a major influence on the numbers.

Other data sets we considered included

• Trends (google, twitter),
• News Articles (page views for news articles),
• Weather (by region),
• Geolocation info could also correlate with that
• Map against custom data (like your own KPI or internal data sets)

This was a very informative project and expect more soon.

Cheers

Team Scallop: Grant Lee, Joel Antipuesto, Matt Thomas, Mekuria Getinet, Paul Krohn

One of my queries dropped an unused schema that I knew was corrupt, so I wasn’t too surprised when the drop database command crashed the MySQL server. After the crash, the server came back up quickly, and my client automatically reconnected, so it was safe to keep running queries right?

Wrong.

When the client reconnected I lost my session state, so sql_log_bin reverted to 1, and any commands I ran from that point forward would be replicated, which I did not want.

This behavior makes sense, and it’s documented in the manual:

Automatic reconnection can be convenient because you need not implement your own reconnect code, but if a reconnection does occur, several aspects of the connection state are reset on the server side and your application will not know about it. The connection-related state is affected as follows:

• Any active transactions are rolled back and autocommit mode is reset.
• All table locks are released.
• All TEMPORARY tables are closed (and dropped).
• Session variables are reinitialized to the values of the corresponding variables. This also affects variables that are set implicitly by statements such as SET NAMES.
• User variable settings are lost.
• Prepared statements are released.
• HANDLER variables are closed.
• The value of LAST_INSERT_ID() is reset to 0.
• Locks acquired with GET_LOCK() are released.

But it’s easy to overlook such details when working with automatic features like MySQL client auto-reconnect. In this specific case I didn’t execute any other commands in the reconnected session so I didn’t inadvertantly replicate anything, but this incident served as a good reminder to be vigilant about my session state when using the MySQL client.

Here’s a snippet from my session to show the value of sql_log_bin before and after the crash:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

mysql> set sql_log_bin = 0;
Query OK, 0 rows affected (0.00 sec)

mysql> select @@sql_log_bin;
+---------------+
| @@sql_log_bin |
+---------------+
|             0 |
+---------------+
1 row in set (0.00 sec)

mysql> drop database test;
Query OK, 1 row affected (0.19 sec)

mysql> select @@sql_log_bin;
+---------------+
| @@sql_log_bin |
+---------------+
|             0 |
+---------------+
1 row in set (0.00 sec)

mysql> drop database reports;
ERROR 2013 (HY000): Lost connection to MySQL server during query
mysql> select @@sql_log_bin;
ERROR 2006 (HY000): MySQL server has gone away
No connection. Trying to reconnect...
Connection id:    505
Current database: *** NONE ***

+---------------+
| @@sql_log_bin |
+---------------+
|             1 |
+---------------+
1 row in set (0.00 sec)

All browsers respect a security specification known as a Same-origin Policy, which means that sites and even iFrames can only communicate with eachother if they share the same origin or host. The document.domain property must match exactly in order for two frames to communicate with each other directly. I say “directly” because methods like using window.postMessage allow for Cross-document messaging which I’ll cover later in the post.

What can you do with document.domain?

Aside from reading the property, you can also set the property using JavaScript, but you may only set it to be less specific. For example, you can change www.flite.com to flite.com, but you cannot change flite.com to www.flite.com. There are some legitimate reasons why a site might make use of changing its document.domain property. One typical use case is for supporting a plugin across a site that has many subdomains. Here’s a fictional example: let’s use flite.com. We have several subdomains where different information is kept for different audiences. There’s the main marketing site www.flite.com, a site for developers at developer.flite.com, and a site to get help at support.flite.com. We also have a widget for serving videos that serves out of video.flite.com which happens to require access to the top frame in order to expand a sharing menu. In order for the video player to work on all three sites, we will either need to inline the video player, enable Cross-Origin Resource Sharing (CORS) on our servers, or we could set the document.domain property in all four of our sites using document.domain = "flite.com". Magically, everything works and we can move along with our lives. This method is all well and good if you are the one controlling each of the sites and can orchestrate this sort of change, but it can wreak havoc on ads and plugins running on your site if they aren’t prepared to handle it.

So, what’s the problem?

Flite Ads are 3rd party and we don’t have control over how sites are implemented. In some implementations, our ads are served in iFrames that share the same domain as the top page where we need to expand into. Under normal circumstances the document.domain property is not changing. In this ideal scenario, we continue to have the ability to expand onto the top page. Recently, we have been seeing a trend among sites where their JavaScript, or some plugin the site is loading, changes the document.domain property of the page. Sometimes it changes before the ad is loaded, sometimes shortly after the ad is loaded, or sometimes not at all. This makes it impossible to always change the document.domain property by default. Instead, our code running inside of the iFrame must try and detect when the parent’s document.domain property changes and update its own in lock-step so that during any future attempts to expand, our code will get the access it may need.

Flite’s secret sauce…

I mentioned 2 scenarios, the first being where document.domain is changed before our ad is loaded. To overcome this, we simply try to determine if we can access the top window’s document object and if not, we change our local document.domain property and try again.

Shifting document.domain using trial and error:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

//Function to check if current window and the top window are Cross-Origin
var isCrossOrigin = function() {
    try{
        //try to access the document object
        if (top.document || top.document.domain){
          //we have the same document.domain value!
        }
    }catch(e) {
      //We don't have access, it's cross-origin!
        return true;
    }
    return false;
};

//Function to shift off the first part of the host name
var shiftDomain = function() {
  var currentDomain = document.domain;
  document.domain = currentDomain.substring(currentDomain.indexOf('.') + 1);
};

//Reduces 2 subdomains down to just the domain only if necessary 
//(e.g. dev.support.flite.com to flite.com)
if (isCrossOrigin()) {
     shiftDomain();   
     
     if (isCrossOrigin()) {
      shiftDomain();
     }
}

The problem with this method is that while it will work and allow you to communicate between frames, all browsers will complain about the first attempt to access the document by throwing an error in the JavaScript console. Even though the code is catching the error, the browser will still report it. It won’t halt execution of subesequent code, but it will cause an error message stating: Unsafe JavaScript attempt to access frame with URL 'blah' from frame with URL 'blah'. Domains, protocols and ports must match.

The second case is what we’ve been encountering more of: the site loads a widget that is loaded after our ad is loaded which requires changing the document.domain property. This bypasses our initial scenario leaving us with a mis-matched document.domain property after we’ve initialized. As soon as this property changes, direct communication between frames in either direction is cut off. As I mentioned earlier, we are able to overcome this. On initialization we create a polling loop using createInterval in the top window that waits until the document.domain property changes from what it once was. Once the change is detected, it clears the interval and executes a call to window.postMessage which sends a message to our frame. We send along the new value in the message and our frame is ready waiting for this message. Upon receiving it, the frame changes the local document.domain property which restores the ability to communicate to the parent frame and vise versa!

Tracking when document.domain changes gracefully:

• JSFiddle showing how when document.domain changes, a child page can no longer access it. The top box is the current frame, the blue box is the child iFrame, both are loaded from the same subdomain:

• The child frame used in the previous example. It displays the current frame’s and the parent frame’s document.domain values:

• JSFiddle showing how a child frame can detect the change and handle it gracefully:

• The child frame with code that tracks document.domain changes in the parent frame:

Final thoughts

If I could have my way, I would push everyone towards abandoning using this property. The reality is that all modern browsers support CORS and we as an industry should be embracing it. It’s one of those things that your developers want to use, but your Operations team likely needs to enable. The Web as a whole moves at glacial speeds when adopting new standards, so while it would be ideal to have everyone move to CORS, it’s not practical. I’ve shown that there are ways to work around it that make your plugin a little more resilient out in The Wild. However, if you develop a plugin or use a plugin on your site that requires changing document.domain, please consider enabling CORS and please stop changing document.domain! As a final note, on Flite.com and all of its subdomains, we do not change the document.domain property, I just needed an example :-).

In this particular case the JSON is pretty simple. It contains two fields: age and gender. Here’s an example of the data format:

1
2
3
4

{"age":"Over 30","gender":"female"}
{"age":"Under 30","gender":"female"}
{"age":"Over 30","gender":"male"}
{"age":"Under 30","gender":"male"}

Parsing that into two separate columns with common_schema is pretty easy; just use the extract_json_value() function like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

mysql> select common_schema.extract_json_value(f.event_data,'/age') as age,
    ->   common_schema.extract_json_value(f.event_data,'/gender') as gender,
    ->   sum(f.event_count) as event_count 
    -> from json_event_fact f
    -> group by age, gender;

+----------+---------+-------------+
| age      | gender  | event_count |
+----------+---------+-------------+
| Over 30  | female  |     3710983 |
| Over 30  | male    |     2869302 |
| Under 30 | female  |     5027591 |
| Under 30 | male    |     4918382 |
| unknown  | female  |       42039 |
| unknown  | male    |       50173 |
| unknown  | unknown |        8372 |
+----------+---------+-------------+

That fits my reporting use case perfectly, and I’m now looking into other ways to use common_schema to parse JSON stored elsewhere in my database.

If you haven’t tried common_schema, I recommend you check it out. It’s easy to install, saves a lot of time and effort, and gets better with each release. In fact, I’m now using 4 of the 7 “New and Noteworthy” features from the 1.3 release.

But there are also some lesser known tools that don’t have anything to do with MySQL.

Here are a few of my favorites:

pt-price

Percona acknowledges that “PT” doesn’t only stand for “Percona Toolkit”, it is also the chemical symbol for Platinum, so they recently added this handy little tool to give you the current spot price for Platinum. The price of Platinum is changing all the time, and it’s hard to keep up with it. Also, currency exchange rates are constantly fluctuating, so figuring out the value of platinum in various currencies is almost impossible. That’s where pt-price comes in. For example, say you want to know the price of Platinum in Euros per kilo or US dollars per ounce right now. Just run one of these commands:

1
2
3
4
5

$ pt-price --currency=euro --unit=kilogram
39586.26

$ pt-price --currency=usd --unit=ounce
1586

pt-height

We all know that the Petronas Towers in Kuala Lumpur are 1483 feet tall, but how many meters is that? How about furlongs? pt-height can tell you:

1
2
3
4
5

$ pt-height --unit=meter
451.9

$ pt-height --unit=furlong
2.247

The Petronas Towers haven’t been the tallest buildings in the world since 2004, but where do they rank right now? pt-height can tell you:

1
2

$ pt-height --rank
7

pt-barnum

I haven’t used this one much yet, so I’ll leave this as an excercise for the reader.

Last week I had to update millions of rows across many databases to tokenize some persisted URL values, and I remembered reading Baron Schwartz’s recent blog post about using the common_schema split feature. Baron’s use case was deleting data, but I figured this could work well to break my large updates into chunks too. I had already written the update statements I wanted to execute, and after five minutes reading the common_schema documentation I was ready to try it out on a dev database.

One of the queries I tried didn’t work in common_schema, but with a creative workaround I was able to trick common_schema into accepting it. Read on for the details.

The first query I wanted to run involved two tables, so I needed to specify which table to split on. In this case I’m updating the parameter table, so I specify that as my split table first, then specify the full update statement, and after that I select $split_total_rowcount so I can see the progress of the update after each chunk is done.

Here’s the code:

1
2
3
4
5
6
7
8
9
10

set @script := "
  split(flite.parameter: update flite.registration
    inner join flite.parameter on parameter.registration_id = registration.id
    set parameter.value = replace(parameter.value,'http://flite.com','${urls.siteUrl}')
    where registration.type in (1,9)
    and parameter.name = 'asset_url'
  ) 
  SELECT $split_total_rowcount AS 'rows updated so far';
";
call common_schema.run(@script);

It worked on dev and QA so I tried it out on my staging db and it worked perfectly, updating over a million rows in about an hour. Here’s the end of the output:

1
2
3
4
5
6
7
8
9

...
+---------------------+
| rows updated so far |
+---------------------+
|             1195047 |
+---------------------+
1 row in set (1 hour 3 min 40.12 sec)

Query OK, 0 rows affected (1 hour 3 min 40.16 sec)

The second query was even simpler, just a single table update. But when I ran this one with common_schema no rows were updated:

1
2
3
4
5
6
7
8
9
10
11

mysql> set @script := "
    ">   split(update flite.registration
    "> set url = replace(url,'http://flite.com','${urls.siteUrl}')
    "> where type in (1,9)
    "> )     
    ">   SELECT $split_total_rowcount AS 'rows updated so far';
    "> ";
Query OK, 0 rows affected (0.00 sec)

mysql> call common_schema.run(@script);
Query OK, 0 rows affected (1.23 sec)

I guess common_schema failed trying to split the table, so I tried a no op call to verify it’s splitting approach, and that returned nothing, which validated my hypothesis:

1
2
3
4
5
6
7
8
9

mysql> call common_schema.run("
    ">   split(flite.registration) { 
    ">     select 
    ">       $split_step as step, $split_columns as columns, 
    ">       $split_min as min_value, $split_max as max_value, 
    ">       $split_range_start as range_start, $split_range_end as range_end
    ">   }
    "> ");
Query OK, 0 rows affected (0.33 sec)

The registration table has over 50 columns, and in addition to having an auto-increment primary key it also has other unique indexes, so of all my tables I’m not too surprised this one had a problem.

So now what do I do? I could submit a bug report for common_schema, or open up the code and try to fix it myself (and I’ll eventually do at least one of those two things), but I really wanted to make this work, so I decided to trick common_schema into doing it anyways. Since common_schema supports multi-table updates, I just rewrote my single table update as a gratuitous multi-table update so I could split on the second table.

First I tried joining to a parent table:

1
2
3
4
5
6
7
8

set @script := "
  split(flite.instance: update flite.registration
inner join flite.organization on organization.id = registration.organization_id
set registration.url = replace(registration.url,'http://flite.com','${urls.siteUrl}')
)
  SELECT $split_total_rowcount AS 'rows updated so far';
";
call common_schema.run(@script);

But when I ran that it didn’t return anything for a few minutes, which was not a good sign. Rather than troubleshoot that query, I decided to try another join table.

Next I tried joining to a child table, using a where clause that I knew would give me a 1-1 relationship with the table I was updating:

1
2
3
4
5
6
7
8
9
10

set @script := "
  split(flite.instance: update flite.registration
inner join flite.instance on instance.registration_id = registration.id
set registration.url = replace(registration.url,'http://flite.com','${urls.siteUrl}')
where registration.type in (1,9)
and instance.is_preview = true
)
  SELECT $split_total_rowcount AS 'rows updated so far';
";
call common_schema.run(@script);

This time it worked! Here’s the end of the output:

1
2
3
4
5
6
7
8
9

...
+---------------------+
| rows updated so far |
+---------------------+
|             1195725 |
+---------------------+
1 row in set (42 min 33.48 sec)

Query OK, 0 rows affected (42 min 33.52 sec)

Now I can go back and look at the code and/or submit a bug report, but I was pleased that with a little help common_schema solved my use case. This was a reminder to me of the power of a good tool, and the value of being creative in the way you use that tool.

Recently I found a perfect use case to start using InnoDB compression on a single table. This table gets a few hundred inserts per day, and each row stores up to 350 KB of serialized JSON in a LONGTEXT column.

Since I use innodb_file_per_table on MySQL 5.5, compressing the table was easy. I just executed 2 lines of DDL:

1
2
3
4
5
6

SET GLOBAL innodb_file_format=BARRACUDA;

ALTER TABLE reg_version 
  ENGINE=InnoDB 
  ROW_FORMAT=COMPRESSED 
  KEY_BLOCK_SIZE=16;

Of course I also added innodb_file_format = BARRACUDA to /etc/my.cnf to make the change permanent.

From my understanding, this change means that any newly created InnoDB tables will use the BARRACUDA format, and all pre-existing tables will still be in ANTELOPE format. I don’t think the format will matter to me unless I want to compress another table, but I’m not sure if rebuilding an existing table via OPTIMIZE TABLE or ALTER TABLE would cause it to be rebuilt in BARRACUDA format or not.

The table is about 80% smaller on disk now that it is compressed, so this change is working as expected for me.

My plan was to use CHECK TABLE on all of the tables to verify the integrity of the backup. The simplest way to run CHECK TABLES on an entire database is like this:

1

mysqlcheck -c --all-databases

Rather than using that command, I decided to generate a bunch of individual CHECK TABLE statements dynamically by running a query on the information_schema.tables table. Why do it that way? It gives me the flexibility to check a specific schema first, or check the smallest or most frequently updated tables first, etc.

So I ran this command to generate my CHECK TABLE script:

1
2
3
4
5

select concat('check table ', table_schema, '.', table_name, ';') as check_stmt
into outfile '/tmp/check_tables.sql'
from information_schema.tables
where table_type = 'BASE TABLE'
order by (data_length + index_length) asc;

And I planned to execute the script like this:

1

mysql -f < /tmp/check_tables.sql

However, when I ran the first query it crashed the MySQL server!

1
2
3
4
5
6

mysql> select concat('check table ', table_schema, '.', table_name, ';') as check_stmt
    -> into outfile '/tmp/check_tables.sql'
    -> from information_schema.tables
    -> where table_type = 'BASE TABLE'
    -> order by (data_length + index_length) asc;
ERROR 2013 (HY000): Lost connection to MySQL server during query

Of course that error message doesn’t always mean that mysqld crashed. Sometimes it just means your session was killed. But in this case I checked, and mysqld did indeed crash.

It was a harmless crash since it was on a passive server, but it pointed out a problem with the backup so I had to find out exactly what went wrong so I could fix it.

I checked the error log and found the source of the crash:

1
2
3
4
5
6
7
8
9
10
11
12

InnoDB: Error: tablespace id is 17722 in the data dictionary
InnoDB: but in file ./test/promo_history_old.ibd it is 18280!
130221  9:39:55  InnoDB: Assertion failure in thread 139984392570624 in file fil0fil.c line 765
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
130221  9:39:55 - mysqld got signal 6 ;
...

I’ve omitted the rest of the error output (including the backtrace) for brevity, but it’s clear which table caused the problem: test.promo_history_old.

After confirming that table is no longer needed I dropped it and re-ran the information_schema query successfully so I could move on with the rest of my integrity check.

In the future I’ll run a quick fail-fast test on information_schema before doing the more time-consuming check of running CHECK TABLE on all of the tables. I think this should be sufficient:

1

select * from information_schema.tables;

Of course I’ll still run CHECK TABLE on all of the tables, but the information_schema test will help me find some problems faster.

Roland Bouman wrote a great post a few years ago that debunks some myths about using GROUP BY in MySQL. His post has a lot of detail and examples, and does a very good job detailing the way GROUP BY works in MySQL with and without ONLY_FULL_GROUP_BY enabled. I recommend that you go and read that post now. Among other things, Roland points out one case where query performance is improved by not using a full GROUP BY. The post is several years old, but the performance difference is still present today in MySQL 5.6.

That post doesn’t make any strong recommendation on using ONLY_FULL_GROUP_BY, so why do I use it? For me, it makes it safer to run dynamically generated report queries. I execute a variety of dynamic reports that select one or more metrics for a specific time period over a specific set of dimensions. In order for the report to return accurate data, all of the dimensional columns must be in both the SELECT clause and the GROUP BY clause. I trust myself to write good SQL by hand, but since I have code generating dynamic SQL I value the extra protection provided by this SQL mode. If MySQL throws an error due to a partial GROUP BY I can catch it with a functional test, rather than trying to catch a more subtle error, namely incorrect report data.

To illustrate the problem I am trying to avoid, I will execute a reporting query on the sakila database. Let’s say I want to see the 10 most popular language/category combinations for the films in the sakila database. I could use a query like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

mysql> select l.name,c.name,count(*)
    -> from sakila.film f
    -> left outer join sakila.film_category fc on fc.film_id = f.film_id
    -> left outer join sakila.category c on c.category_id = fc.category_id
    -> left outer join sakila.language l on l.language_id = f.language_id
    -> group by c.name
    -> order by count(*) desc
    -> limit 10;
+---------+-------------+----------+
| name    | name        | count(*) |
+---------+-------------+----------+
| English | Sports      |       74 |
| English | Foreign     |       73 |
| English | Family      |       69 |
| English | Documentary |       68 |
| English | Animation   |       66 |
| English | Action      |       64 |
| English | New         |       63 |
| English | Drama       |       62 |
| English | Games       |       61 |
| English | Sci-Fi      |       61 |
+---------+-------------+----------+
10 rows in set (0.00 sec)

That looks pretty good. Now imagine my query was written by a query builder in code, and due to a bug the category column was not added to the GROUP BY clause. By default MySQL will still execute the query and give me a result, but the result is misleading:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

mysql> select l.name,c.name,count(*)
    -> from sakila.film f
    -> left outer join sakila.film_category fc on fc.film_id = f.film_id
    -> left outer join sakila.category c on c.category_id = fc.category_id
    -> left outer join sakila.language l on l.language_id = f.language_id
    -> group by l.name
    -> order by count(*) desc
    -> limit 10;
+---------+-------------+----------+
| name    | name        | count(*) |
+---------+-------------+----------+
| English | Documentary |     1000 |
+---------+-------------+----------+
1 row in set (0.01 sec)

That makes it look like every film in the sakila database is a Documentary, which is obviously not accurate.

To avoid this I will set the SQL_MODE. When setting the SQL_MODE you need to be careful not to remove any existing SQL_MODE values. In my case I will append ONLY_FULL_GROUP_BY to the 2 SQL_MODE values I am already using:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

mysql> select @@sql_mode;
+--------------------------------------------+
| @@sql_mode                                 |
+--------------------------------------------+
| STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION |
+--------------------------------------------+
1 row in set (0.00 sec)

mysql> set session sql_mode = 'STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION,ONLY_FULL_GROUP_BY';
Query OK, 0 rows affected (0.00 sec)

mysql> select @@sql_mode;
+---------------------------------------------------------------+
| @@sql_mode                                                    |
+---------------------------------------------------------------+
| ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION |
+---------------------------------------------------------------+
1 row in set (0.00 sec)

Now that I’ve set the SQL_MODE, I will try my query again:

1
2
3
4
5
6
7
8
9
10

mysql> select l.name,c.name,count(*)
    -> from sakila.film f
    -> left outer join sakila.film_category fc on fc.film_id = f.film_id
    -> left outer join sakila.category c on c.category_id = fc.category_id
    -> left outer join sakila.language l on l.language_id = f.language_id
    -> group by l.name
    -> order by count(*) desc
    -> limit 10;
ERROR 1055 (42000): 'sakila.c.name' isn't in GROUP BY
mysql> 

That’s much better. In my case I would rather get an error than get bad data, so this is the result I want.

Running LOAD DATA INFILE commands at the mysql prompt gives you pretty good output, but if you run the same command at the terminal or in a shell script you have to do a little bit of extra work to capture the errors and warnings.

Here are a few techniques I use when I run LOAD DATA INFILE at the terminal or in a shell script:

1. Use double-verbose mode (--v) to capture the high level counts of Records, Deleted, Skipped, Warnings, etc.
2. Use --init-command to set session variables (sql_mode, sql_log_bin, foreign_key_checks, etc)
3. Use --show warnings to get all of the warnings, even if they exceed the value of max_error_count
4. Use user-defined variables and SET statements to explicitly handle NULL values and defaults.

Below I will provide more detail on each of those techniques, using the sakila database for my code examples:

1. Use double-verbose mode (--v) to capture the high level counts of Records, Deleted, Skipped, Warnings, etc.

For my first example, I’ll dump all of the rows from a table with a primary key into a flat file on disk, and then try to load that flat file into the table again and see what happens.

If I do a straightforward LOAD DATA INFILE I get an error:

1
2
3
4
5

mysql> select * into outfile '/tmp/rental.txt' from sakila.rental;
Query OK, 16044 rows affected (0.02 sec)

mysql> load data infile '/tmp/rental.txt' into table sakila.rental;
ERROR 1062 (23000): Duplicate entry '1' for key 'PRIMARY'

If I add the IGNORE keyword I can avoid that error, and MySQL tells me that all 16044 rows were skipped as duplicates:

1
2
3

mysql> load data infile '/tmp/rental.txt' ignore into table sakila.rental;
Query OK, 0 rows affected (0.23 sec)
Records: 16044  Deleted: 0  Skipped: 16044  Warnings: 0

The IGNORE keyword is also useful if your file contains some duplicates and some new rows. To illustrate this I delete 1000 random rows from the table and then re-run the same command:

1
2
3
4
5
6

mysql> delete from sakila.rental where rental_id >= 8000 and rental_id < 9000;
Query OK, 1000 rows affected (0.11 sec)

mysql> load data infile '/tmp/rental.txt' ignore into table sakila.rental;
Query OK, 1000 rows affected (0.74 sec)
Records: 16044  Deleted: 0  Skipped: 15044  Warnings: 0

The 1000 rows I deleted were successfully loaded back into the table, and the other 15044 rows were rejected as duplicates.

By default, running LOAD DATA INFILE from a shell script will not give you any output unless there is an error.

To illustrate what happens in a shell script I’ll run the same commands from the terminal.

Error conditions are the same:

1
2
3
4

$ mysql --database=sakila \
>   -e "load data infile '/tmp/rental.txt' into table sakila.rental;"
ERROR 1062 (23000) at line 1: Duplicate entry '1' for key 'PRIMARY'
$ 

If I add the IGNORE keyword, by default I don’t get the information about how many rows were skipped:

1
2
3

$ mysql --database=sakila -e \
>   "load data infile '/tmp/rental.txt' ignore into table sakila.rental;"
$ 

I can capture that information by passing in the --verbose flag, but I need to pass it twice, and I get some extra information, specifically the SQL command I execute is echoed back to me:

1
2
3
4
5
6
7
8
9
10
11

$ mysql --database=sakila \
>   -vve "load data infile '/tmp/rental.txt' ignore into table sakila.rental;"
--------------
load data infile '/tmp/rental.txt' ignore into table sakila.rental
--------------

Query OK, 0 rows affected (0.43 sec)
Records: 16044  Deleted: 0  Skipped: 16044  Warnings: 0

Bye
$ 

2. Use –init-command to set session variables (sql_mode, sql_log_bin, foreign_key_checks, etc)

--init-command is a handy way to set your session up the way you want it when running mysql commands from the terminal or in a shell script. For example, if your server uses a relaxed global sql_mode, you may enforce a stricter sql_mode for your session like this:

1

mysql --init-command="set sql_mode = 'STRICT_ALL_TABLES';"

You can set multiple variables in a single init-command. For example you can set the sql_mode, disable binary logging for your session, and disable foreign key checks like this:

1

mysql --init-command="set sql_mode = 'STRICT_ALL_TABLES'; set sql_log_bin = 0; set foreign_key_checks = 0;"

To further illustrate this point, I’ll use the sakila.rental table again in several different error scenarios that benefit from the use of --init-command.

First I’ll truncate the table before I load the data back in:

1
2
3

$ mysql --database=sakila \
>   -e "truncate table sakila.rental;"
ERROR 1701 (42000) at line 1: Cannot truncate a table referenced in a foreign key constraint (`sakila`.`payment`, CONSTRAINT `fk_payment_rental` FOREIGN KEY (`rental_id`) REFERENCES `sakila`.`rental` (`rental_id`))

Oops, I can’t truncate the rental table because the payment table has a foreign key to it. Since I’m planning to load the data back into the rental table I want to just ignore the foreign key. I can do that with --init-command:

1
2
3
4

$ mysql --database=sakila \
>   --init-command="set foreign_key_checks=0;" \
>   -e "truncate table sakila.rental;"
$ 

That’s better. Now I’ll change return_date from NULL to NOT NULL and see what happens when I try to load the flat file back in, knowing that it has 183 rows with NULL values in that column:

1
2
3
4
5
6
7
8
9
10
11
12
13

$ mysql --database=sakila \
>   -e "alter table sakila.rental modify column return_date datetime not null;"
$ mysql --database=sakila \
>   -vve "load data infile '/tmp/rental.txt' into table sakila.rental;"
--------------
load data infile '/tmp/rental.txt' into table sakila.rental
--------------

Query OK, 16044 rows affected, 183 warnings (0.82 sec)
Records: 16044  Deleted: 0  Skipped: 0  Warnings: 183

Bye
$ 

All of the rows loaded successfully, but I got 183 warnings. Unfortunately I don’t see any of the warnings, which brings me to:

3. Use --show warnings to get all of the warnings, even if they exceed the value of max_error_count

I’ll try that truncate and load again using --show-warnings so I can see all of the warnings.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

$ mysql --database=sakila \
>   --init-command="set foreign_key_checks=0;" \
>   -e "truncate table sakila.rental;"
$ mysql --database=sakila \
>   --show-warnings \
>   -vve "load data infile '/tmp/rental.txt' into table sakila.rental;"
--------------
load data infile '/tmp/rental.txt' into table sakila.rental
--------------

Query OK, 16044 rows affected, 183 warnings (0.82 sec)
Records: 16044  Deleted: 0  Skipped: 0  Warnings: 183

Warning (Code 1263): Column set to default value; NULL supplied to NOT NULL column 'return_date' at row 11492
Warning (Code 1263): Column set to default value; NULL supplied to NOT NULL column 'return_date' at row 11537
Warning (Code 1263): Column set to default value; NULL supplied to NOT NULL column 'return_date' at row 11559
Warning (Code 1263): Column set to default value; NULL supplied to NOT NULL column 'return_date' at row 11573
Warning (Code 1263): Column set to default value; NULL supplied to NOT NULL column 'return_date' at row 11589
Warning (Code 1263): Column set to default value; NULL supplied to NOT NULL column 'return_date' at row 11607
Warning (Code 1263): Column set to default value; NULL supplied to NOT NULL column 'return_date' at row 11642
...
Bye
$ 

I snipped the list of warnings for brevity. It was 183 occurrences of the same warning.

This begs the question, what default value did MySQL use?

1
2
3
4
5
6

$ mysql --database=sakila -e "select min(return_date) from sakila.rental;"
+---------------------+
| min(return_date)    |
+---------------------+
| 0000-00-00 00:00:00 |
+---------------------+

That’s not ideal. I’ll try again with a stricter sql_mode that disallows zero dates:

1
2
3
4
5
6
7
8
9
10
11
12
13

$ mysql --database=sakila \
>   --init-command="set foreign_key_checks=0;" \
>   -e "truncate table sakila.rental;"
$ mysql --database=sakila \
>   --show-warnings \
>   --init-command="set sql_mode = 'STRICT_ALL_TABLES';" \
>   -vve "load data infile '/tmp/rental.txt' into table sakila.rental;"
--------------
load data infile '/tmp/rental.txt' into table sakila.rental
--------------

ERROR 1263 (22004) at line 1: Column set to default value; NULL supplied to NOT NULL column 'return_date' at row 11492
Bye

Okay, that’s better. I don’t get any zero dates, but instead none of the data is loaded. Which brings me to:

4. Use user-defined variables and SET statements to explicitly handle NULL values and defaults

Let’s say I don’t want zero dates, but I do want to enforce a default value for rows without a date in the specific context of my LOAD DATA INFILE statement. For the purposes of this example I’ll use the current date and time as returned by the MySQL NOW() function.

I can capture the relevant column values from the flat file in a user-defined variable, and then use a SET statement within LOAD DATA INFILE to explictly set return_date=now() if a NULL is in the file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

$ mysql --database=sakila \
>   --show-warnings \
>   --init-command="set sql_mode = 'STRICT_ALL_TABLES';" \
>   -vve "load data infile '/tmp/rental.txt' into table sakila.rental \
>   (rental_id,rental_date,inventory_id,customer_id,@return_date,staff_id,last_update) \
>   set return_date = coalesce(@return_date,now());"
--------------
load data infile '/tmp/rental.txt' into table sakila.rental   (rental_id,rental_date,inventory_id,customer_id,@return_date,staff_id,last_update)   set return_date = coalesce(@return_date,now())
--------------

Query OK, 16044 rows affected (0.99 sec)
Records: 16044  Deleted: 0  Skipped: 0  Warnings: 0

Bye
$ 

Let’s take a simple example:

You want to bind a click handler to each LI element of a UL in the DOM. Let’s say you decide on doing it by getting a reference to the items, loop over them, and bind a click handler that will alert the value of the loop counter “i”. You might expect that each item will alert “0”, “1”, “2”, etc. right? Well, if you run the example, you will see that it is not the case! Turns out that because of how JavaScript scopes things, the reference to “i” is held in each click handler function and is updated as it is incremented in the loop. So instead of getting a nice list of items that happily alert the appropriate index number, you get a list of items that all alert “5”. Annoying!

I’m not saying you would actually try to code things as I did in the example, but what are some other ways to do the same thing that would avoid this problem? Here are just a few other examples that you might consider:

1. Bypass using the loop entirely

jQuery makes it super easy to just skip the whole looping mess using the power of CSS selectors and the on() function. Without a loop or counter, we can use jQuery’s index() function to alert the index as we were trying to do in the first example too. try it

2. Use function closures

The first example works, but what if you still want to do things using a loop? Since the problem stems from the variable reference problem, using closures will safely protect the variable values and properly give you the expected result. try it

3. Use a separate function

If you don’t like using closures for whatever reason, defining a separate outside function also achieves the same result. This approach might even be a bit cleaner and/or reusable. try it

What are some ways you have solved this type of problem? I’d love to hear from our readers to see how they might have done things differently!

For further reading on this topic, I found this article rather interesting.

The most obvious reason to use unique server_id values for every database in a replication setup is to avoid having a master and slave with the same server_id. If that happens the slave will skip any events coming from that master.

But if I have two slaves with the same server_id, that should be safe unless I promote one of those slave to be the master of the other. Is that the case that MySQL is trying to protect me from, or is there some other case I am missing?

My specific problem started when I cloned a slave database. This is something I do fairly often, but most of the time I am creating a new read slave, and I change its server_id to something unique. This time I was replacing an existing database, so I left the server_id the same. When I started the slave on the new copy of the database, I now had two slaves with the same server_id. It didn’t seem like a big deal at the time, because I knew I would be shutting down the old copy in the near future.

The first symptom I noticed was an explosion of relay logs on the new database. On a healthy slave database there should usually only be one or two relay logs, but this database had tens of thousands of them, all very small, and all very recent.

I looked into the MySQL error log to see what was going on, and saw the same error over and over again:

1
2
3
4
5
6
7
8
9
10
11
12

130122 20:28:08 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'bin-log.000049' at position 340959933
130122 20:28:08 [Note] Slave: received end packet from server, apparent master shutdown: 
130122 20:28:08 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'bin-log.000049' at position 340959933
130122 20:28:08 [Note] Slave: received end packet from server, apparent master shutdown: 
130122 20:28:08 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'bin-log.000049' at position 340959933
130122 20:28:08 [Note] Slave: received end packet from server, apparent master shutdown: 
130122 20:28:08 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'bin-log.000049' at position 340959933
130122 20:28:09 [Note] Slave: received end packet from server, apparent master shutdown: 
130122 20:28:09 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'bin-log.000049' at position 340959933
130122 20:28:09 [Note] Slave: received end packet from server, apparent master shutdown: 
130122 20:28:09 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'bin-log.000049' at position 340959933
130122 20:28:09 [Note] Slave: received end packet from server, apparent master shutdown: 

I did a search for that error message and found Peter’s post from a few years ago which pointed me in the right direction.

So I changed the server_id on the old slave db, restarted it, and the errors went away.

In the future I’ll be more careful about this when cloning databases, but I have to say this seems like an odd way to enforce this restriction, and not a very descriptive error message. Even though the master was constantly sending end packets to the slave and trying to make it go away, the slave continued to reconnect and process updates from the master, although it certainly did so inefficiently.

I’ve learned my lesson for the future, but I’m still left with two questions:

1. Why is MySQL enforcing unique server_ids for slave databases?
2. Can we get a more descriptive error message when this situation occurs?

The ExternalInterface class is extremely handy, but troubleshooting your implementation can prove to be difficult. Here are some helpful tips for using the ExternalInterface class that I’ve picked up along the way.

Basic examples of using ExternalInterface

The most useful function of the ExternalInterface class is call. A typical use case is to execute existing JavaScript that might be on the page. But a more exciting use case is to execute your own JavaScript functions in ActionScript code. Anonymous functions are great for one time usage. For example, opening a predetermined URL:

1

ExternalInterface.call("function(){window.location.href = 'http://flite.com';}");

ExternalInterface.call also allows you to pass any number of parameters to JavaScript functions. The first argument is the function (passed as a String), followed by the parameters to be passed to the function. For example, you can get trace output on machines without the Flash Debugger version installed. ExternalInterface can be used to log into your browser’s JavaScript debugging console instead of the default location.

1

ExternalInterface.call("console.log", myVar, myVar2);

For times when you need to make repeated calls to the same function, improve your performance by using static constants.

1
2

private static const ALERT_FN:String = "function(msg){alert(msg);}";
ExternalInterface.call(ALERT_FN, "ding!");

As your code gets more complex, I suggest breaking it up onto several lines to make it readable.

1
2
3

private static const ALERT_FN:String = "function(msg){" +
    "alert(msg);" +
"}";

How to Debug your JavaScript

Make your code debuggable by adding “debugger;” statements. The newline characters can help with formatting of the code when the breakpoint catches in the debugging console. Most developer tools – such as Chrome’s Web Inspector – allow you to format JavaScript, so the newline characters may not be needed.

1
2
3
4

private static const ALERT_FN:String = "function(msg){\n" +
    "debugger;\n" +
    "alert(msg);\n" +
"}";

Improve performance and your ability to debug your code even further by injecting complex functions onto the page in an initialization function. By doing this, you can also execute your injected JavaScript functions directly in the browser’s debugging console. I recommend creating a namespace object on the window that is unique to your SWF, so that you don’t overwrite existing code in JavaScript.

1
2
3
4
5
6
7
8
9
10
11

private static const MYFUNC_FN:String = "window.MY_NAMESPACE.myFunc";
private static const INIT_JS:String = "window.MY_NAMESPACE = window.MY_NAMESPACE || {};" +
MYFUNC_FN + " = function(arg1, arg2){ /*do a lot of things*/}";

public function MyConstructor(){
     ExternalInterface.call(INIT_JS);
}

private function doMyFunc(p1:String, p2:Boolean) : void {
     ExternalInterface.call(MYFUNC_FN, p1, p2);
}

From JavaScript to ActionScript

Beyond just executing javascript, Flash can also evaluate return values coming back from JavaScript. You could use JavaScript to generate random numbers, for example:

1
2
3

private function getJSRandom() : Number {
     return Number(ExternalInterface.call("Math.random"));
}

Avoid passing objects that reference non-primitive types (i.e. DOM, Functions, etc). JavaScript is designed to work with the DOM object which is known to be full of circular references. For example, a DOM node has a property referencing its parent element and the parent also has references to its children. When passing values back and forth between Flash and JavaScript, the arguments and return values are serialized and the circular references will cause Flash to come crumbling to its knees.

For example, the first code block will not work, while the second one will:

1
2

var el:Object = Object(ExternalInterface.call("document.getElementById('container')");
trace(el.nodeName);

1
2

var nodeName:String = String(ExternalInterface.call("function(){return document.getElementById('container').nodeName;}");
trace(nodeName);

And finally, if the need arises, you can also call into your ActionScript code from JavaScript by using the ExternalInterface.addCallback function to register an ActionScript function. JavaScript can also evaluate return values returned from your ActionScript functions.

1
2
3
4
5

//AS3 Code
var doSomething:Function = function(arg1) : void{
    //do something here
}
ExternalInterface.registerCallback("myASFunc", doSomething);

1
2
3
4
5

//JS Code
var myObj = document.getElementById("my-object-tag");
if(myObj.myASFunc) {
   myObj.myASFunc("foo");
}

For more on Flash’s ExternalInterface class, see Adobe’s API documentation here.

Here’s the problem that inspired me to make this change.

It started when one of our passive MySQL master database hosts restarted unexpectedly. The host came back up fairly quickly, and MySQL started up cleanly once InnoDB finished its crash recovery.

However, all of this master’s slave databases were in a failed state with the same error:

1
2

Last_IO_Errno: 1236
Last_IO_Error: Got fatal error 1236 from master when reading data from binary log: 'Client requested master to start replication from impossible position'

Apparently some replication events were lost from the end of the binary log when the host went down, so the slave was asking for a position that no longer existed. Since this was a passive master we didn’t lose any data (the relevant events were still on the active master and came through in the next binary log on the passive master), but replication from this master was temporarily broken.

I manually rolled all of the slaves forward to the appropriate binary log using CHANGE MASTER, but they were behind for several minutes before I did that.

Digging through the configuration on the master I discovered that sync_binlog was disabled, as it is by default in MySQL 5.5. Here’s how that option is explained in the manual:

If the value of this variable is greater than 0, the MySQL server synchronizes its binary log to disk (using fdatasync()) after every sync_binlog writes to the binary log. There is one write to the binary log per statement if autocommit is enabled, and one write per transaction otherwise. The default value of sync_binlog is 0, which does no synchronizing to disk—in this case, the server relies on the operating system to flush the binary log’s contents from to time as for any other file. A value of 1 is the safest choice because in the event of a crash you lose at most one statement or transaction from the binary log. However, it is also the slowest choice (unless the disk has a battery-backed cache, which makes synchronization very fast).

That’s a pretty good description of the trade-offs involved, and given my write volume I figured it would be worth it for us to enable that option. I did some testing to verify that our masters can handle the extra IO overhead and then enabled those settings on all master databases where it was disabled. Now replication will be more likely to resume automatically after a master failure.

I also updated all of our slave databases to enable sync_relay_log and sync_relay_log_info so if a slave db restarts it’s more likely that replication will resume automatically. As expected, the manual enumerates the same “safest” and “slowest” arguments for sync_relay_log.

The manual is less clear about how to set sync_master_info and sync_relay_log_info. It recommends disabling sync_master_info and enabling sync_relay_log_info, even though both are disabled by default. And it doesn’t explain the reasoning behind those recommendations. I guess that’s left as an exercise for the reader :)

In my opinion, if you enable sync_relay_log it makes sense to enable sync_master_info and sync_relay_log_info, too.

While there’s a well-documented workflow to interacting with npm packages (npm install, npm test, npm publish, rinse and repeat) there are several powerful npm commands that are very useful but less commonly used.

1. The power of npm scripts

It’s generally known that the scripts member of a package.json file allows developers to specify actions that npm can execute. For instance, if you have a scripts member in your package.json file that looks like this:

1
2
3
4

"scripts": {
    "start": "node server.js",
    "test": "grunt test"
}

The command npm start will execute node server.js and npm test will execute the grunt test command (or any other shell script). It’s very useful for defining a common interface to start and test node.js packages across projects.

What’s less well-known, however, is that the scripts member supports over dozen other actions that can execute before and after other npm commands. For example:

1
2
3

"scripts": {
    "preinstall": "./bin/custom-script.sh"
}

Before npm install is run, the ./bin/custom-script.sh file is executed. If the custom script returns a non-zero exit code, the npm install command aborts.

Other actions include postinstall, poststart, prepublish and pretest.

This is very nice for coffeescript projects. You can define a prepublish script that compiles your *.coffee files to *.js and your app is ready to run in node.js. All users have to do is run npm install as they normally would.

Alternatively, you could have prestart script that uses grunt to minify and concatenate *.js files before starting the web server. This is particularly useful for avoiding including minified and optimized resources in source control, not to mention making it easier to deploy to cloud providers like heroku and nodejitsu.

2. Faster package development with npm link

npm link is an essential tool when developing npm packages that you’d like to test in other projects on your system. It creates a globally-installed symlink for a package folder and then allows you to use that globally-installed symlink in another project. As you change the linked package, the other package that gets the changes instantly. It’s great for package development.

As the documentation states, it’s a two step process:

• Run npm link in a package folder (typically the package you’re developing).
• In other package folder (typically the one you want to test with your in-development package) run npm link <your development package name>.

That’s it.

3. Keeping track of old dependencies with npm outdated

The npm package world moves very fast. It’s easy to write a new package and then have its dependencies get out-of-date. The npm outdated command does exactly what it says – it reads your package.json file and tells you which of your dependencies have new versions.

It’s a simple idea but one that I’ve tried to work into my development workflow more often. It’s a good sanity check to make sure I have the fairly recent versions installed of my package’s dependencies.

4. Know your npm aliases

How many times have you typed npm install after cloning a git repository? That’s an entire 11 characters! Fortunately, there’s a shortcut: npm i does exactly the same thing.

The list of aliases is longer than you might expect – there’s no less than four ways you can uninstall a package: npm rm, npm unlink, npm r, npm remove and npm un.

There’s even a helpful npm isntall alias that maps to npm install.

Here’s the complete list.

5. Shell completion with npm completion

npm aliases are nice, but what’s even better than aliases is not typing text at all. npm completion is one of those commands I wish I knew about months ago – it prints a shell script snippet that adds shell completion for npm commands.

For example, typing npm i<tab> then gives you all of the wonderful npm command possibilities that start with i:

1
2

sh $ npm i [press tab]
i        info     init     install

To install npm shell completion, it’s just a matter of adding the script npm completion outputs to your shell init file. Or, even better, add it to your dotfiles repository.

npm has great man pages, and with autocompletion it’s even easier to learn new interesting commands.

Bonus: Happy Holidays from npm!

Finally, perhaps the one command every node.js developer should know about: npm xmas

Open your terminal and run this command immediately.