Big data analytics can help decision-makers turn large, complex datasets into patterns they can inspect and use. A study of crime records from San Francisco, Chicago and Philadelphia illustrates a practical workflow: prepare records for analysis, visualize their geographic and temporal distribution, then compare forecasting models. Although the application is public safety, the same analytical principles are relevant to industrial teams working with incident, maintenance, quality or operations data.
Big data analytics starts with usable data
The study examined millions of publicly available incident records collected across different periods in the three cities. The datasets included timestamps, incident categories, approximate addresses, coordinates and other attributes. Such volume creates an opportunity for analysis, but raw records are not automatically ready for comparison or forecasting.
Preprocessing converted timestamps into usable time fields, including year, month, day, hour and minute. The researchers also removed fields not needed for their analysis and addressed missing location coordinates in the Chicago and Philadelphia records. Their approach imputed missing coordinates using values sampled from available coordinates and then calculated a mean to replace missing entries.
This stage matters in any analytics pipeline. Data quality decisions influence what patterns can be found and how confidently results can be interpreted. For industrial applications, teams should document transformations, missing-value treatment and exclusions so that analysts and operational stakeholders understand what the dataset represents.
Visualization connects time and location
To make the geographic dimension easier to explore, the study used an interactive map that grouped incident markers at broader zoom levels and displayed individual locations at closer views. This approach made it possible to move between a city-wide view and more localized patterns without presenting every record as a separate marker at once.
Time-based charts added another perspective. Annual, monthly and hourly summaries revealed changes that would be difficult to see in unaggregated records. The analysis reported shared hourly patterns across the cities, with fewer recorded incidents around 5â6 a.m. and a peak around midnight. Monthly patterns varied: Chicago and Philadelphia showed higher incident counts in summer, while the timing of peaks differed in San Francisco.
These findings are descriptive, not proof of why the patterns occurred. The paper discusses possible relationships with factors such as population density, climate and activity levels, but the reported patterns alone do not establish causation. That distinction is important when presenting operational dashboards: a visible correlation can guide further investigation, but should not be treated as an explanation without supporting evidence.
For industrial users, the equivalent might be mapping downtime by production area, plotting defects by shift or comparing alarms by time of day. Visualization helps teams identify where and when to investigate, while keeping the underlying limits of the data visible.
Comparing models for time-series forecasting
After summarizing incidents per day, the researchers tested several approaches to forecast trends. These included a conventional neural network, a Long Short-Term Memory (LSTM) model and Prophet, a time-series procedure that represents observations through trend, seasonality, holiday effects and residual error.
Prophet is designed to model changes over time alongside recurring patterns, such as weekly or yearly seasonality. LSTM is a recurrent neural network architecture suited to sequences in which earlier observations may help explain later ones. The conventional neural network provided a comparison with those approaches.
Model evaluation used Root Mean Square Error (RMSE) and Spearman correlation. RMSE measures the size of prediction errors, while Spearman correlation indicates how closely the predicted and observed values move in rank order. Using more than one measure provides a broader view than relying on a single score: a model may have relatively small errors but still track changes poorly.
In the tested datasets, Prophet and the stateful LSTM performed better overall than the conventional neural network. The experiments also found that three years of training data produced the best results among the training-window sizes they compared. More historical data did not automatically improve forecasts. This is a finding for these datasets and experiments, not a universal rule for other applications.
What the results mean for operational analytics
The studyâs most useful lesson for technical decision-makers is methodological: forecasting depends on the fit between the data, the model and the evaluation process. A forecasting project should not assume that the most complex model or the longest history will be best.
- Define the time scale. The analysis aggregated records by day for trend forecasting. Industrial teams should choose a resolution appropriate to the question and the way events are recorded.
- Make data preparation explicit. Timestamp conversion, missing values and field selection can affect results, so these decisions need to be understood before model comparisons begin.
- Compare suitable measures. Error magnitude and correlation describe different aspects of prediction quality. Evaluate models against the intended operational use.
- Use visualizations to investigate, not just report. Maps and time-series plots can help teams examine anomalies, recurring patterns and differences between locations or operating periods.
- Treat forecasts as decision support. The study presents predictions as input for resource planning, not as certainty about individual future events.
The paper also identifies further analytical directions, including multivariate visualization, graph mining and more detailed spatial analysis. These extensions reflect a broader challenge: patterns in large datasets may involve multiple variables and relationships that simple summaries cannot capture.
Conclusion: build an evidence-led analytics workflow
This case study shows how big data analytics can combine careful preprocessing, interactive visualization and time-series forecasting to make large datasets more interpretable. Its results suggest that Prophet and LSTM were stronger choices than a conventional neural network for the examined crime trends, while a three-year training window performed best in the tested comparisons. For industrial automation and operations teams, the practical takeaway is to validate data and models in context, make assumptions transparent, and use analytical outputs to informânot replaceâsound operational judgment.
Access Official Technical PDF