Blog
Blog
Web blog of the DASH project group
Hello and a warm welcome to our blog.
10 June 2016
CEWE Foundation & Co. KGaA invited us to the CEWE Analytics Forum, which took place on 7 June. For this event, we prepared a presentation specifically tailored to an audience comprising members of the Executive Board and department heads. The project group’s findings to date were well received by the audience.
On 8 and 9 June, a nationwide, interdisciplinary conference for student research took place at the University of Oldenburg. The event is called ‘forschen@studium’. Here, students can present their research findings from seminar papers, dissertations or project work. Further information on the conference can be found at uol.de/lehre/qualitaetspakt-lehre/flif/forschen-at-studium/konferenz-fuer-studentische-forschung/.
To take part in the conference, all interested participants had to summarise their research in an abstract of one page in length. We, too, wrote an abstract, which was accepted and published alongside the other accepted abstracts in the conference proceedings of ‘forschen@studium’. Our abstract can be found on page 106 (uol.de/fileadmin/user_upload/flif/Konferenz2016/Broschur_Forschen_A4_Internet.pdf). We are also aiming for a publication of our work in the student online journal ‘forsch!’.
At the conference in Oldenburg, we presented our findings in the form of a poster during a poster session. This gave us the opportunity to inform a number of interested parties about our project. We also received positive feedback.

Source: Photograph by Simon Becker
16 May 2016
In our ELT process, following AXT (Agent for XML Transportation), the PreX (Prepare XML) component had been planned to merge the XML files and then pass the merged files to the parser. However, we have now decided not to implement the PreX functionality as a separate component, but to incorporate it into AXT.
Furthermore, there remains the challenge of parsing large volumes of data using Hadoop Streaming and Spark. For this reason, some members of the analytics team will now support the parser team and drive this work forward.
CEWE Foundation & Co. KGaA has offered us the opportunity to present our findings to date at the CEWE Analytics Forum. We intend to accept this offer and are currently preparing a presentation that will focus less on the technical details and more on the results that can be achieved through our work.
Coinciding with this intensive phase of development, the date for the project group boules tournament has now been announced. We will, of course, be taking part. To boost team spirit, we have also organised a football match against the DOHA project group.
2 May 2016
In software development, it is often the case that as soon as one bug has been fixed, the next one crops up. We have found this to be true of our ELT process. Our AXT (Agent for XML Transportation) is now running flawlessly, as the faulty method has been replaced by another. However, we are now experiencing difficulties when parsing the data. We are experiencing out-of-memory issues with both the Hadoop Streaming variant and the Spark alternative, meaning that so far we have only been able to parse small amounts of data.
To ensure that the other objectives can still be achieved, part of the project team is already familiarising themselves with the XML files with a view to data understanding. Initial analyses of the existing data are even already being carried out.
In addition, we have now appointed test coordinators within our project group to take charge of test management. To this end, test scripts for the individual components will be drawn up initially.
From an organisational perspective, we want to restructure our approach. So far, we have been meeting every Thursday for the whole day so that we can work intensively on the project, either as a group or in sub-groups. This time spent together is very important because it allows for a great deal of discussion and consultation. However, we often find ourselves short of 1–2 hours during which organisational matters are dealt with, such as the weekly scrum, sprint retrospective and sprint planning. We now intend to move this organisational part to another day so that we can be even more productive on Thursdays.
For our second social event, we went to the Barcelona Finca.
18 April 2016
Sometimes, in day-to-day project work – particularly within a student-organised group – set targets cannot be achieved despite careful planning and coordination.
For instance, our goal of completing a full ELT run could not be met by the set milestone due to issues within AXT (Agent for XML Transportation). In-depth research revealed that when transferring larger files, the FTP server returned an error code. This error was not processed correctly by the method used from the Apache commons.net library, resulting in a kind of infinite loop.
However, to ensure that subsequent milestones are not affected, the project group is simultaneously exploring requirements and ideas for initial analyses and hypotheses.
It also became apparent that, when using Git to manage our data, separate repositories are required for the various components. This is due to the requirements of the individual components and their growth. This has resulted in a new structure within our Git: AXT (Agent for XML Transportation), Parser, HIMP (Hive Importer; formerly Watchdog), Airflow, Tests, Analytics, Documentation, Miscellaneous.
Furthermore, in future we must get into the habit of making changes to the source code on a branch first. Only once the changes have been successfully tested can they be pushed to the master branch. This ensures that the master branch is always stable and contains a working version.
4 April 2016
Over the past two weeks, the ‘Dashboard and Visualisation’ sub-group has been evaluating tools as part of the software selection process. The decision on which tool to use for visualisation and for our dashboard was made in favour of Apache Zeppelin. It meets the specified requirements and is easy to use.
The other sub-groups are working to meet the deadlines for the milestone on 7 April 2016, which involves running the entire ELT process (see diagram showing the four steps).

- In the first step, the XML data is transferred from the FTP server to HDFS using our in-house application AXT (Agent for XML Transportation). HDFS features a tree-like folder structure organised by year, month and day.
- The XML data is then prepared. This means either that several XML files are combined into a single XML file and stored in the ‘ready2parse’ folder within the HDFS, or that the original XML files are copied into the ‘ready2parse’ folder. This step is carried out using our own tool, PreX (Prepare XML).
- The XML files from the ‘ready2parse’ folder are parsed. The results can then be found in the ‘parsedData’ folder in HDFS for the fourth step.
- Finally, in the last step, the data is loaded into Hive tables using Watchdog. Watchdog has been customised for this application and does not correspond to a ‘standard Watchdog’, as monitoring in the strict sense is not carried out.
The entire workflow is controlled using the Airflow workflow management system.
18 March 2016
In addition to the two existing sub-groups for the ELT process, two new sub-groups have now been set up to complete outstanding tasks within the project group:
- Dashboard and Visualisation sub-group
- Data Mining Subgroup
There are therefore currently four sub-groups, each comprising three people, within the PG.
No suitable application could be identified for loading the XML data into HDFS, so the decision was taken to implement this task in-house: ‘Agent for XML Transportation’ (AXT). Unfortunately, it only became apparent in retrospect, during the evaluation of individual functionalities rather than at an earlier stage, that no tool exists which meets the requirements and that development must be carried out in-house.
Regardless of the implementation, one of the PG’s objectives is to gain knowledge of the methods and tools used, which became apparent at the end of the seventh sprint. During this sprint, it was noted that Confluence offers the option to create a retrospective, which will be used from now on to improve future sprints.
The current plan includes the milestone of 7 April 2016: running the ELT process on the alpha version. To this end, the tools we have developed in-house, such as AXT or the parser, must be complete and operational by then. In addition, all components involved must be linked together via a workflow management tool.
29 February 2016
The first status meeting of 2016 with CEWE Foundation & Co. KGaA takes place. The current status of the two sub-groups and the component selection to date are presented. CEWE Foundation & Co. KGaA provides us with the sizing specifications for the Hadoop environment for the beta version. It is also planned that the project group will give a presentation in June at an internal CEWE event, the so-called Analytics Forum.
- The Gobblin tool is still being investigated. Apache Spark is being investigated as an alternative to Hadoop Streaming. The parser can be called within Apache Spark.
- The Python parser can be used to parse the existing XML files. The results are saved in CSV files in accordance with the designed database model. However, the implementation is still static at this stage. A dynamic (partial) solution is to be in place by the beta version at the latest.
- Importing the CSV files into the Hive tables works. We are now assessing whether the Apache Oozie workflow tool is suitable for checking whether new CSV files are available.
- There are plans to set up a logging system. This will help to identify, in the event of errors, in which component or at which step the error occurred.
Programming guidelines are being drawn up. Although we have already started programming, it is not too late to do this. The source code produced so far is still manageable and can be easily revised.
To ensure that the project does not end up with a purely documentation-focused phase, the results achieved so far are already being compiled into a document for the final report.
Each team member is beginning to familiarise themselves with the topics of data mining and visualisation, or dashboards, so that the upcoming tasks can be properly understood and allocated. Two new sub-groups will be formed shortly. One will then focus on data mining, whilst the other will deal with visualisation and dashboards.
The project group’s first social event took place at the 3Raumwohnung. There, we played table football, chatted and simply had a good time.
12 February 2016
About halfway through the project group’s term, the first feedback meetings are taking place. Each team member has a one-to-one meeting with the supervisors to find out their current grade or progress, and to raise any issues or specific concerns.
As we realised last time that our use of JIRA isn’t yet running smoothly, we’re now carrying out a spring clean of our JIRA system. Some team members are taking on this task. This involves revising the epics (thematic areas in JIRA) and renaming the various statuses so that a clear distinction can be made between ‘in testing’ and ‘complete’. A task is only considered ‘complete’ once it has been tested.
Hadoop Streaming is being investigated. This makes it possible to distribute Python scripts across the cluster. As an alternative, the manual distribution of scripts is being explored. The parser itself is being further developed. In addition, Hive tables are being created. The Gobblin tool for extracting and loading data is being investigated. As this tool is still relatively new and under development, there are a few issues with it.
An initial social event is planned for the end of the month. We expect to hold these regularly in future.
30 January 2016
The two server administrators from our group are installing the Hadoop ecosystem on the servers at the University of Oldenburg for our alpha version.
With regard to extracting and loading the data, the tools Apache Kafka, Apache Flume, Logstash and Fluentd are being evaluated in terms of their installation and configuration requirements, as well as their compatibility with other tools. It has now become apparent that Fluentd and Logstash are not suitable for our project. Data can be transferred from the FTP server to HDFS using Apache Flume. However, there are some issues with the configuration of Apache Flume. Apache Kafka is still being investigated. Some team members have discovered a new tool called Gobblin and are investigating it. As a backup solution, one team member has also written a script themselves, which can transfer the XML files from the FTP server to HDFS without any problems.
It has been decided to use a custom-built parser to parse the XML files. The Python programming language is being used for this. Alongside the development of the parser, a relational data structure is already being designed. This is to be used for creating Hive tables. The plan is for the parser to generate corresponding CSV files, which will be imported into the Hive tables.
Our supervisors have pointed out that Apache Hadoop does not perform as well with many small files as it does with larger ones. We will therefore attempt to combine several small files into one large one. We have also been advised to use Hadoop Streaming and MapReduce to parse the XML files. We are looking into this as well.
We have also noticed that, although we intend to work according to SCRUM, we have not yet held a proper sprint review or retrospective for any of the sprints so far. Our sprints last two weeks and we are already on our fourth sprint. We intend to change this at the end of the current sprint.
13 January 2016
Following the Christmas holidays, we are continuing to investigate which of the various components of the Apache Hadoop ecosystem are best suited to our project. In doing so, we are examining the following components in detail:
For extracting and loading the data, we are evaluating the tools Apache Kafka, Elasticsearch, Logstash, Fluentd, Apache Storm and Apache Flume. With Apache Storm, we have encountered configuration issues relating to Apache ZooKeeper. The configuration of Apache Storm and Nimbus was successful. Elasticsearch is not suitable for our requirements. The tools Logstash, Apache Flume and Fluentd have relatively similar performance characteristics, which is why we will now compare the installation and configuration effort as well as compatibility with other tools.
To transform and parse the data, we are examining the tools Apache Hive, Apache Pig and Apache Spark. Apache Spark is being considered in conjunction with the various programming languages Java, Scala and Python. So far, it has become apparent that Apache Pig is unsuitable for our purposes, as it does not allow individual attributes to be extracted from the XML file. Apache Hive and Apache Spark can both be used to parse the data for our purposes. However, as Apache Spark is easier to configure and use, we have decided to go with this option.
In addition to the component selection, part of our group is currently working intensively on the XML files. The aim is to produce documentation to aid understanding of the XML files. At present, we do not yet know what each individual element and attribute represents, or what information lies behind them. To be able to determine this in future, documentation is being drawn up and discussed with staff from our industry partner. This forms a necessary basis for the future analysis of the data.
Alongside these activities, we have noticed that our current organisational structure has some room for improvement. For instance, we are not yet making full use of JIRA, as evidenced by the fact that tasks are not always assigned correctly and the information in the system is generally not kept up to date. Often, the person responsible for development has also tested the results themselves. As a result, proper testing of the results has not really taken place up to now. Now, immediately after a task is completed, it is assigned to another person who is also familiar with the subject matter of the task for testing. This means the person who carried out the task receives more feedback. We aim to achieve better results in this way. For the tool to be truly effective, we must take greater care to keep the tasks up to date and assign them to the right people. Only then can it have a positive impact on our work outcomes.
Outlook for 2016
Having begun familiarising themselves with the subject matter in 2015, having identified the requirements, and having concluded the year with an evaluation of the Apache Hadoop components, the project group will continue in 2016 with the development of an initial prototype.
Overall, 2016 is divided into two phases as far as the project group is concerned. The first phase runs until April and involves the development of an initial prototype (alpha version). The second period runs from April to September and covers the development of the second prototype (beta version) as the outcome of the project group’s work. The main difference between the two versions is that the alpha version runs on servers within the University of Oldenburg, whilst the beta version is integrated into the system environment of CEWE Foundation & Co. KGaA.
In the alpha version, we still have the opportunity to try everything out and test it. We can install all components and evaluate them from a wide variety of perspectives. And that is exactly what will be happening a great deal. In April, our system will then be moved from the servers at the University of Oldenburg to the servers at CEWE Foundation & Co. KGaA. This is where the beta version will be deployed. The components we ultimately intend to use should therefore be finalised by April, as no components are to be tested during the beta version.
We have planned to begin analysing the data from March onwards. We distinguish between two forms of analysis: analytics on the one hand, and data mining on the other. By ‘analytics’, we mean the pure determination of key performance indicators. By ‘data mining’, we refer to all analytical topics that are more complex.
However, in order to be able to analyse the data at all, we must extract the XML data, load it into the Hadoop Distributed File System (HDFS) and transform the data. We began work on this in December 2015 and will continue with it throughout January and February 2016.
2015 Year in Review
The project group got underway in October 2015. The first order of business was to get to know the other project group members and the supervising lecturers. We then met our contacts at CEWE Foundation & Co. KGaA and were introduced to the DigiFotoMakers (DFM for short). The DFMs are kiosk terminals belonging to CEWE Foundation & Co. KGaA, which are installed at their retail partners’ premises, such as Müller Großhandels Ltd. & Co. KG. Photo products can be ordered and printed directly on site using these terminals. When a DFM is operated, technical data as well as information on user behaviour is saved in an XML file. Our task is to be able to analyse this XML file using Apache Hadoop.
Before we could get down to the actual work on our project, we had to agree on how we wanted to organise ourselves over the coming year. We very quickly settled on SCRUM as our software development methodology, as we are all familiar with it, know the process well and feel comfortable using it. To improve organisation, we chose Jira along with its associated documentation tool, Confluence. We also use Git for file sharing. Furthermore, a number of roles were assigned; we now have two project managers, two server administrators, a documentation manager and a website manager.
Once the organisational matters had been sorted out, the requirements were drawn up and defined in detail. In doing so, we identified three core requirements, which are briefly mentioned here. A detailed description can be found under ‘Project’.
- ‘Near Time’ analysis of key performance indicators
- An alarm system that detects failures and downtime
- Analysis of user behaviour to optimise the ordering software
As the next step, we assessed which components of the Apache Hadoop ecosystem we would use. To this end, the project group split into two sub-groups. The first sub-group examined components for interface extraction and loading data into the Hadoop Distributed File System (HDFS). The second sub-group focused on components for transforming the XML files. The traditional ELT process was thus split into two parts. At the last meeting before the Christmas holidays, the individual sub-groups presented the interim status of their findings.
10 December 2015
As a first step in Hadoop, we want to decide on the required components. Because Hadoop is very extensive and comprises many different but similar components, we need to test some of them and examine them in more detail. A scientific approach is particularly important here in order to be able to make a decision. To this end, the project group has split into two sub-groups. The first sub-group is focusing on components for interface extraction and data loading. The second sub-group is focusing on components for transformation strategies. The ELT process has thus been divided.
Once we have made the selection of components, we intend to move on to the next phase. This means we will then focus on the topic of analytics, which includes data mining and machine learning.
We plan to coordinate the final selection of the components to be installed with CEWE Foundation & Co. KGaA. Although we will initially install everything within the university, we plan to migrate the architecture to the system landscape of CEWE Foundation & Co. KGaA in spring 2016.
17 November 2015
After the DASH project group had spent several weeks examining potential use cases and business cases in detail, the specific objectives were agreed with the CEWE Foundation & Co. KGaA supervisors. The following objectives were defined:
- Alarm system: This term has caused some confusion, which is why a precise definition in our glossary was urgently required. It does not simply refer to a ticketing service that issues a message when a warning or alarm is triggered. By this, we also mean the logic required to detect a warning or alarm in the first place.
- Operational dashboard: This serves, above all, to provide the project group with access to the specific XML data. The dashboard provides an overview of certain key figures, such as products ordered per day.
- Clickstream analysis: By ‘clickstream’, we mean the anonymised log of clicks made within the Digi-Foto-Maker ordering software. Analysing this data enables us to improve the user-friendliness of the Digi-Foto-Maker. It is also possible to identify whether certain paths consistently lead to abandoned transactions or whether certain combinations are frequently selected.
The focus of our work will be more on developing an ETL process and on data management in Apache Hadoop. For CEWE Foundation & Co. KGaA, it is less important exactly which data we analyse or how we present the results visually.
To improve our understanding of the XML data, a workshop in collaboration with CEWE Foundation & Co. KGaA is planned for January. We will also be given access to a Digi-Foto-Maker, on which we can create some test cases ourselves. Furthermore, CEWE Stiftung & Co. KGaA has invited us to give a presentation at the so-called ‘CEWE Analytics Forum’. This is an internal CEWE Stiftung & Co. KGaA event at which our project would even be presented to the Executive Board.
25 October 2015
As well as answering the question ‘Where are we heading?’, we have also begun to consider how we actually intend to get there over the course of the next year. To this end, the project group has agreed on a number of organisational points. In some cases, these agreements were not always entirely clear and there was a great deal of back-and-forth discussion.
Where we reached agreement very quickly was on adopting a Scrum-based approach, guided by CRISP DM. In both cases, we will tailor the approach to our specific needs. With regard to Scrum, we have currently agreed on fortnightly sprints. Our weekly meeting will serve as a ‘Weekly Scrum’ rather than a ‘Daily Scrum’.
As a further organisational matter, we have decided to use Jira and Confluence. Jira will be used for task management and Confluence for documentation. We are also setting up a GIT repository, although there are still a few uncertainties as to what purpose this is intended to serve, given that we already have Confluence. At least we don’t need to document things in both systems at the same time.
To lighten the mood a bit, we’ve arranged a photo shoot. You can view the photos under ‘The Team’. This will give you an idea of who the PG DASH team and ‘we’ actually are. There’ll be another photo shoot in January, when we’ll also take a group photo with the supervisors from both the University of Oldenburg and CEWE Foundation & Co. KGaA.
After some back and forth and plenty of creative brainstorming, we’ve now settled on our PG name, ‘DASH’ (Data Analytics with Hadoop), and our logo.
18 October 2015
In the first few weeks, we’re in the process of identifying the exact requirements. This is proving not to be as straightforward as we thought. To begin with, we were given only a broad objective. Otherwise, our client, CEWE Stiftung & Co. KGaA, is giving us a great deal of freedom to work as we see fit.
To determine the direction we should take, we will set out specific objectives. To this end, we have first considered possible use cases. We have also reflected on the requirements, both from a technical perspective regarding the system and from a user’s perspective.
We’re planning a meeting with CEWE Foundation & Co. KGaA for next week. We intend to present our findings so far and discuss with our client which use cases from our longlist should be implemented where possible. Furthermore, a number of questions have arisen during our deliberations so far, which we still need to resolve.
As well as working on the requirements, everyone is still writing their seminar papers. We will be presenting these to one another shortly. The results will also be available from December under ‘Seminar Papers’.
13 October 2015
In the second week, we visited CEWE Foundation & Co. KGaA. There, we were introduced to our contacts and shown the photo stations, also known as Digi-Foto-Maker (DFM for short). The DFMs are the self-service kiosks operated by CEWE Foundation & Co. KGaA, which are installed at retail partners such as Müller. These machines allow customers to place photo orders or have photo products printed out immediately on the spot. When a DFM is operated, user behaviour and technical data relating to the DFM – such as the printer’s temperature – are stored in an XML file.
Our task for the coming year is to be able to analyse these XML files using Apache Hadoop. To achieve this, we must first familiarise ourselves with Apache Hadoop and set up such a system. In addition to this introduction to the subject, CEWE Foundation & Co. KGaA has already provided us with initial requirements and possible analysis topics that we should implement over the course of the project. This involved a ‘near-time’ analysis: an alert system to detect failures and downtime, and to further optimise the ordering software through a detailed analysis of user behaviour.
4 October 2015
The first project group meeting took place on campus with the project group participants and the supervising lecturers, Andreas Solsbach and Viktor Dmitriyev. Everyone introduced themselves briefly and organisational matters were clarified.
For the second meeting, we then tried to find a fixed weekly time that suited everyone. To this end, we created a Doodle poll. However, it is very difficult to agree on a time with 12 people plus 2 lecturers. Furthermore, a room at the university must also be available at that time. Unfortunately, we couldn’t avoid the fact that some participants had to make compromises. As we students all work part-time, we looked at the times we still had available when nobody had to be at work. We also took into account our lecturers’ request that the session should take place sometime between 8 am and 4 pm. In the end, we were able to reach an agreement. Nevertheless, a few students will now be unable to attend other lectures or even tutorials as a result.
1 October 2015
We are the DASH project group at the University of Oldenburg.
Our project group (PG) is called DASH – Data AnalyticS with Hadoop – because we will be spending a year working with Apache Hadoop. We are collaborating with the company CEWE Stiftung & Co. KGaA, which is known, amongst other things, for its photo services. You can find more details on this in the ‘The Project’ section.