Developer Contest Now Open: Tell Us Your Story
Have you downloaded HPCC Systems and done something cool with it?
Tell us your story for a chance to win prizes like Google Glasses! Developers and other technical professionals have the opportunity to demonstrate how they leveraged HPCC Systems to solve either a Big Data or Complex Query problem. HPCC Systems® is the company’s open-source, enterprise-proven platform for big data analysis and processing for large volumes of data in 24/7 environments.
"We look forward to learning more about the experiences developers and other companies are having by leveraging HPCC Systems to solve their unique challenges," said Flavio Villanustre, Vice President, Products and Infrastructure, HPCC Systems, LexisNexis. "To entice the technical community to enter the case study contest, we are offering some cool prizes, including Google Glass Explorer Edition technology for first place, a Go Pro Hero3+ Silver Edition Video Camera for second place, and Bose® IE2 Audio Headphones Ear buds for the first top 10 successful case study submissions."
HPCC Systems underpins the data services of LexisNexis Risk Solutions, with revenues of $1.46 billion. LexisNexis creates data, risk, analytics and anti-fraud solutions for customers in financial services, insurance, government, health care and retail.
The U.S.-only contest is now open and runs through Feb. 27, 2015. Entrants must meet case study requirements and acknowledge the different rules as provided at firstname.lastname@example.org. More information can be found here. First and second place winners will be announced by Mar. 30, 2015.
Judges will evaluate submissions based on the following criteria:
- overall completeness of case study (50% weighting);
- ability to demonstrate a "before" and "after" story (20% weighting);
- complexity of the original problem (10% weighting); and
- include an architecture, diagram or data flow (20% weighting).
We look forward to hearing from you!
Back to Top
Use Case Example
This section discusses how HPCC Systems is used to solve various business challenges.
Using LexisNexis Identity Analytics and HPCC Systems to verify and authenticate the identities of customers applying electronically for public assistance benefits, the Florida Department of Children and Families (DCF) has saved more than $12 million in cost avoidance by preventing fraud and creating increased efficiencies within its first five months rolling out a statewide program.
For its efforts, DCF was recently named by Florida Governor Rick Scott as a "Governor’s Savings Award" recipient.
The DCF has embarked on an aggressive campaign to eliminate fraud, waste and abuse in public assistance programs, utilizing LexisNexis and the company’s big data platform, HPCC Systems, to verify and authenticate the identities of customers applying electronically for public assistance benefits.
"We are extremely pleased with the savings we’ve already realized in the short time our program has been implemented," Office of Public Benefits Director Andrew McClenahan said. "We look forward to continuing to protect the identity of our clients and stopping fraud at the front door, making sure only those who truly need help receive it."
"By verifying and authenticating the identity of the customer before processing the application, DCF isn’t just fighting fraud; it’s actually preventing fraud," said Haywood Talcove, Chief Executive Officer, Government, LexisNexis. "We congratulate DCF for being honored with this award, being the first social services agency in the nation to take this proactive approach to preventing fraud and – most importantly – for ensuring public assistance benefits go to those who truly deserve them."
The Governor’s Savings Award "honors those who answer the Governor’s call for increased efficiencies and cost savings in government operations." LexisNexis identity management solutions are integrated into Automated Community Connection to Economic Self Sufficiency (ACCESS) Florida to help quickly confirm that customers are who they say they are before processing their benefit applications.
See more here.
Back to Top
ECL Tips and Tricks
Enterprise Control Language (ECL) is the programming language of HPCC Systems, and is used to program both the data processing jobs on the HPCC Systems Data Refinery engine (Thor) and the queries on the HPCC Systems Data Delivery engine (Roxie). ECL is a declarative, implicitly parallel and data flow oriented programming language that abstracts complex data processing tasks by providing a simple programming interface.
Below are two tips for a Group Join, a SMART join and an Embed. More tips can be found in the HPCC Systems Redbook.
New Group Join
The new GROUP JOIN syntax allows you to efficiently join two datasets on one condition, but have the result grouped by another condition. This is useful for efficiently solving some relationship matching problems. As a first approximation the following ECL:
R := JOIN(l, r, LEFT.key = RIGHT.key, t(LEFT,RIGHT), GROUP(leftId))
where leftId is a value assigned from LEFT.Id inside the transform t() is equivalent to:
DL = DISTRIBUTE(L, HASH(key));
DR = DISTRIBUTE(R, HASH(key));
SL = SORT(DL, id, LOCAL);
JR := JOIN(SL, DR, LEFT.key = RIGHT.key, t(LEFT,RIGHT), LOOKUP MANY, LOCAL);
DJ := DISTRIBUTE(J, HASH(LEFTID), MERGE(LEFTID));
R := GROUP(DJ, LEFTID, LOCAL);
A new flexible lookup join - JOIN, SMART
A SMART join attempts to perform an in-memory LOOKUP join. If there is insufficient memory, smart join will automatically ensure that both sides are efficiently distributed and attempt to perform a LOCAL LOOKUP join.
If there is still insufficient memory, smart join will become a LOCAL HASH join which is not limited by memory.
Embedded Language Features
It is now possible to pass records and datasets into embedded functions, and to return them as results from embedded functions. Moreover, only those records which are actually required will be evaluated. Also now includes additional languages including MySQL. For more information see: https://track.hpccsystems.com/browse/HPCC-10455
Back to Top
"How To" Demo: Sort and Deduplication
The featured video below explains how to perform a sort and deduplication
Check out the demo
Back to Top
HPCC Systems RoadMap Update:
5.0 Version Now Available!
This section discusses updates and enhancements to the platform.
Enhancements include: With community contributions from developers and analysts across the globe, HPCC Systems is offering translations and localization in its version 5.0 for languages including Chinese, Spanish, Hungarian, Serbian and Brazilian Portuguese with other languages to come in the future.
Additional enhancements include:
- Linux Ubuntu Juju Charm Support
- Embedded language features
- Apache Kafka Integration
- New Regression Suite
- External Database Support (MySQL)
- Web Services-SQL
The HPCC Systems source code can be found here
The HPCC Systems platform can be found here:
Back to Top
About HPCC Systems
HPCC Systems is the platform developed and leveraged by LexisNexis Risk Solutions, which is $1.4 billion information solutions company and a division of Reed Elsevier, a $9 billion global publisher and information provider operating in the science, medical, legal, risk and business sectors.
Back to Top
Adventures in GraphLand Part 1:
HPCC Systems in the News
Click on this link to read a discussion from LexisNexis Chief Technology Officer Vijay Ragahavan on how data can be used to create social good.
The HPCC Systems team will be speaking or appearing at these events: Full Calendar
Looking to further your knowledge on a proven Big Data platform?
See what one of our customers had to say about our training: "I would simply say it's a new and beautiful journey for me. I try to find ways of using this platform in solving various Big Data problems. I'm deeply happy that there's the machine learning library and linear algebra. Also, the ability of embedding python and other language in ECL is just awesome. Still on the journey!
— Itauma Isong Itauma, Wayne State University (Detroit, Michigan)
Training Option 1;
Free Online Introductory Classes
Check out this link for our free Online Introductory ECL training.
Training Option 2:
Advanced classes are available and require a course fee:
Training Option 3:
Of course, if you prefer a face-to-face training class with an experienced instructor in a comfortable and productive setting, check out our classroom training here.
Training Option 4:
Customized Training Plan
on-site or via Webex:
Custom training at your location or via Webex is also available for large groups. Contact us at email@example.com for more information.
Click here for the full training schedule
Have a question or a
tip to share?
Join our forums to start a conversation or check out the job opportunities for ECL developers.