The Best Network Performance Metrics Monitoring Tools

Network Performance, as simple as it may seem, car be a complex topic. Most administrators define it as “the thing users always complain about” and while this is probably true, it is more complicated than that. Today, we’re having a look at network performance metrics. They are the factors that combine to affect how your users perceive the performance of the network. The great thing about those metrics is that, as opposed to user perception—which can be difficult to evaluate, they can be objectively measured, This is why they’re called metrics in the first place.

Network Performance Metrics

Since we want to make sure that everyone is on the same page for the bulk of our discussion, we’ll start off by describing network performance. We will then carry on and discuss the different measurable factors and parameters that can affect the performance of a network. These are we normally refer to as network performance metrics. And once we’re done with the theory, we’ll have a look at some of the best network performance metrics monitoring tools.

About Network Performance

There are several ways one can define performance. In a nutshell, network performance is the subjective perception that the network is performing as expected. In other words, a network’s performance is a measure of its ability to meet its users’ expectations. What makes the matter more complex is that, although performance is highly subjective, measuring it requires a knowledge of the various metrics that do combine to affect perceived network performance. In additions, some usages of the network have very high-performance requirements (VoIP, for instance) while others will be fine on low-performance networks.

ALSO READ: 15 Best Network Monitoring Tools

Network Performance Metrics

Many factors can have an effect on network performance. And fortunately, some of them can be monitored or measured relatively easily. Let’s have a look at what factors and parameters are interacting to give users the perception of good—or no so good—performance. Some of these factors are physical characteristics of networks over which we don’t have much control while others are elements that can often be improved, thereby giving users the perception of better performance.

Bandwidth and Throughput

Speed, bandwidth and throughput are different concepts despite a lot of confusion between them. Let’s try to sort it out, starting with bandwidth. Bandwidth refers to the maximum amount of data that can be transferred per unit of time. It is, in other words, the data-carrying capacity of a network and although circuits can be upgraded, this is a complex endeavour and as such, bandwidth is not considered to be something we can easily control “on the fly”. It is also not something we need to measure as it doesn’t change over time or with increased usage. It’s always the same.

As for throughput, it refers to the actual amount of data carried on a network. Measured throughput is typically not equal to bandwidth, unless in situations of network congestion. It is important to monitor throughput as when it gets too close to the available bandwidth (typically over 70%), network performance starts being impacted.

Much of the confusion between bandwidth and throughput stems from the fact that most “bandwidth monitoring tools” are actually monitoring throughput. They don’t measure bandwidth but rather bandwidth USAGE. The key here is that throughput or bandwidth usage is one of the most important metrics to keep an eye on.

Delay and Latency

Delay and latency are two of the main factors affecting perceived network performance. Much like bandwidth and throughput, there is some confusion between them to the point that the two concepts are often used interchangeably. This is understandable as both have to do with the time it takes for data to travel from its source to its destination. Latency is often described as the time elapsed from the moment the source sends a packet to the moment it arrives at its destination. It can also refer to the round-trip delay time which comprised the one-way latency from source to destination plus the one-way latency from the destination back to the source. In fact, round-trip latency is a more common metric as it can be measured from a single point.

FURTHER READING: Best Network Diagram Mapping and Topology Software

Like Bandwidth, latency is a physical characteristic of networks. It is a factor of the distance between the source and the destination and the speed at which data travels over the media. Like bandwidth, latency is a fixed parameter. The only way to reduce it is to move the source closer to the destination. Reducing the distance by 100 km (about 60 miles) will remove about 1 millisecond of latency.

Several factors can add delay to network transmissions. For instance, queuing delay occurs when a gateway receives multiple packets from different sources heading towards the same destination. Since only one packet can be transmitted at a time, some of them must be queued for transmission, incurring an additional delay. Likewise, processing delays are incurred while a gateway—such as a router—determines how to handle a newly received packet. Buffering can also cause increased delays of an order of magnitude or more. The combination of propagation, queuing, and processing delays often result in a complex and variable network latency profile.

Jitter

Jitter is the biggest enemy of network transmission. Understanding how and why it can have such an adverse effect on data transmissions can be somewhat complicated. Let’s try to explain. Simply put, jitter is a variation in delay. There are several factors that can cause jitter. In fact, many of the same delay-causing factors also increase jitter. For example, queuing delays are directly related to queue length. And since a typical queue constantly varies in length, so does delay, hence jitter.

But jitter does not affect all network traffic in the same way. The risk that is posed by jitter is that, if delays vary considerably between the multiple packets that compose a message, they could arrive at their destination out of sequence. Let’s take, for example, a transmission comprised of four packets that are transmitted at 10 ms intervals. The first one encounters 20 ms of latency, the second one 60 ms, the third one 40 ms and the last one 20 ms. I’ll spare you the boring math but, in this specific case, the first packet will arrive first, followed by the fourth, then the third and finally the second. In many situations, this isn’t a problem. For example, if we’re dealing with a file transfer, the packets are sequentially numbered and can easily be reassembled in the proper order at the receiving end. On the other hand, if what we have is real-time traffic such as a VoIP conversation, packets cannot be correctly reassembled in real-time, resulting in garbled audio. From a user’s standpoint, we’re having a performance issue.

Packet Loss

Packet loss is another major factor affecting perceived network performance. Networks are not perfect and, from time to time, data packets don’t make it to their destination for various reasons. When this happens to TCP traffic, it is not much of an issue as the receiving end can request a retransmission of the missing data. It will, however, cause some delay and increase the data volume potentially affecting performance, especially if it happens a lot. This is much worse with UDP traffic. A lost packet will be lost forever. In a VoIP conversation, that would result in audio dropouts which, if too severe, can render speech unintelligible. In both cases, it is very clear that packet loss will have an impact on network performance.

RELATED READING: Best IP Scanners for Mac (Our own review)

What are the best network performance metrics monitoring tools?

So, to keep a watchful eye on network performance, what you need is some type of monitoring tool that can read all the important metrics and display them in an understandable and “easy-to-digest” way. This is what we’ve looked for, and we’ve come up with a handful of monitoring tools which are among the best ones you can find. Some of these tools might need some efforts beyond the standard setup to get them to monitor all the required metrics but rest assured that these efforts will be worth their while.

1. SolarWinds Network Performance Monitor (Free Trial)

SolarWinds is one of the very best makers of network administration tools. The company is famous for its excellent network administration tools and for its numerous free tools. Its flagship product is called the Network Performance Monitor, or NPM. It is a very complete network monitoring solution. It features a user-friendly graphical user interface that administrators can use to monitor devices and to configure the tool.

Adding a new device to the Network Performance Monitor is as simple as specifying its IP address or hostname and SNMP connection parameters. The system then queries the device and lists all the available metrics. All you have to do is pick those you want to monitor. And talking about adding devices, this tool’s scalability is another one of its best features. It will suit the smallest of networks and scale up to large networks with thousands of devices spread over multiple sites.

SolarWinds NPM - Network Summary

The SolarWinds alerting system is another place where the product shines. It is highly customizable when needed but it can also be used out-of-the-box with minimal configurations. The alerting engine is smart enough not to send notifications for “unimportant” events in the middle of the night or to send hundreds of notifications for as many unresponsive servers when the main issue is a down router or network switch.

Pricing for the SolarWinds Network Performance Monitor starts at just under $3 000 and goes up according to the number of devices to monitor and the selected optional components. The pricing structure is quite complex and you should contact the SolarWinds sales team for a detailed quote. If you prefer to try the product before purchasing it, a free 30-day trial version is available for download from the SolarWinds website.

2. PRTG Network Monitor

The PRTG Network Monitor, which is often simply called PRTG, is another great monitoring system. Its publisher claims that this tool can monitor all systems, devices, traffic, and applications of your IT infrastructure. It is an all-inclusive package that does not rely on external modules or add-ons that need to be downloaded and installed. Because of its integrated nature, it is quicker and easier to install than most other network monitoring tools. You can choose between a few different user interfaces such as a Windows enterprise console, an Ajax-based web interface, and mobile apps for Android and iOS.

PRTQFG Dashboard - Datacenter Monitoring

The PRTG Network Monitor is different from most other monitoring tools in that it is sensor-based. Various monitoring features can be added to the tool simply by configuring extra sensors. They are like plugins except that they are not external modules but are, instead, included with the product. PRTG includes over 200 such sensors that cover different monitoring needs. For network performance metrics, the QoS sensor and the Advanced PING Sensor allow you to monitor latency and jitter while the standard SNMP sensor will let you monitor throughput.

The PRTG pricing structure is pretty simple. There’s a free version which is full-featured but will limit your monitoring ability to 100 sensors. There’s also a 30-day trial version which is unlimited but will revert back to the free version once the trial period is over. If you want to keep monitoring more than 100 sensors beyond the trial period, you’ll need to purchase a license. Their price varies according to the number of sensors from $1 600 for 500 sensors to $14 500 for unlimited sensors. Each monitored parameter counts as one sensor. For example, monitoring bandwidth on each port of a 48-port switch will count as 48 sensors.

3. ManageEngine OpManager

ManageEngine is another well-known publisher of network management tools. The OpManager is a complete management solution that will handle pretty much any monitoring task you can throw at it. The tool runs on either Windows or Linux and is loaded with great features. Among others, there is an auto-discovery feature that can map your network, giving you a uniquely customized dashboard.

The ManageEngine OpManager‘s dashboard is super easy to use and navigate, thanks to its drill-down functionality. And if you are into mobile apps, there are apps for tablets and smartphones allowing you to access the tool from anywhere. This is an overall very polished and professional product.

ManageEngine OpManager Dashboard

Alerting is just as good in OpManager as are all its other components. There is a full complement of threshold-based alerts that will help detect, identify, and troubleshoot network issues. Multiple thresholds with different notifications can be set for all network performance metrics.

If you want to try the product before buying, a free version is available. Although it is a truly free version rather than a time-limited trial, it has some limitations such as letting you monitor no more than ten devices. This is insufficient for all but the smallest of networks. For larger networks, you can choose between the Essential or the Enterprise plans. The first will let you monitor up to 1,000 nodes while the other goes up to 10,000. Pricing information is available by contacting ManageEngine’s sales.

4. WhatsUp Gold

WhatsUp Gold from Ipswitch is another well-known name in the field of monitoring tools. Once an up-or-down type of monitoring, it has since evolved into a full management toolkit with proactive monitoring for network, applications, virtual environments, and device configurations. Today, WhatsUp Gold has everything we’ve come to expect from an enterprise-grade monitoring tool, all available through an intuitive GUI.

WhatsUp Gold features an auto-discovery engine that will find your devices and add them to the monitoring console. It will not only find your networking equipment but also physical servers, virtual servers, cloud servers, and applications. There’s also a map view that’s clickable for more information on each device.

WhatsUp Gold - Home Dashboard

WhatsUp Gold also has an excellent alerting system to let you know about problems before users call. Through the tool’s Alert Center, you can opt to use out-of-the-box thresholds or adjust them to your specific needs. The system allows you to create action policies to define what happens when a state change occurs. Alerts can be transmitted by email, SMS, Slack, or IFTTT posts. The system can also restart services and trigger web alarms.

There’s also an easy-to-use plug-in that collects data and reports on the quality of service (QoS) levels of your network. It uses data generated by Cisco IP SLA-enabled devices to monitor network performance metrics such as jitter, latency, and packet loss,

A free edition of WhatsUp Gold is available–as it always was–although it is now limited to monitoring a maximum of five devices. For more devices than that, paid licenses are available in three levels of increasing functionality with a pricing structure based on the number of devices to be monitored. There’s also a free, full-featured trial version that you can use for a limited time.

5. Zabbix

Zabbix is a free and open-source product which has a highly professional look and feel, much like you’d expect in a commercial product. But the good looks of its user interface is not its only asset. The product also boasts an impressive feature set. Zabbix will monitor most network-attached devices in addition to networking equipment. It would be a good option if you want to monitor servers in addition to your WAN circuit’s bandwidth.

Zabbix Dashboard Screenshot

 

Zabbix uses SNMP as well as the Intelligent Platform Monitoring Interface (IMPI) for monitoring devices. You can use the software to monitor bandwidth, device CPU and memory utilization, general device health as well as configuration changes. The tool is also easily customizable through the use of scripts that will let you configure advanced monitoring. Such scripts exist to monitor most network performance metrics. It also features an impressive and completely customizable alerting system. It will not only send email or SMS alerts but also run local scripts which could be used to fix some issues automatically.

Read The Best Network Performance Metrics Monitoring Tools by Renaud Larue-Langlois on AddictiveTips – Tech tips to make you smarter

6 Best Log Management Tools For Linux in 2019

With today’s systems generating a ton of logging data, it’s no surprise that administrators are always looking for log management solutions. Logs are, by default, often stored locally. This makes sense as it makes it easy to link them to their source. But when trying to troubleshoot issues and find their root cause, we sometimes have to look at multiple log files on numerous devices. Wouldn’t it be nice if all the logs from all devices were stored in one, centralized place? This is the purpose of log management. And if your platform of choice is Linux, there are plenty of options available. Read on as we discover some of the best log management for Linux

Log Management Tools For Linux

We’ll start off by defining log management. You will see that it can be quite a bit more than just centralizing log storage. Next, we’ll discuss various logging technologies. They are the cornerstone of log management and it wouldn’t likely exist without them. Continuing, we’ll differentiate syslog servers from log management systems and realize that there is no clear demarcation between them. Next, we’ll pause briefly and discuss Security Information and Event Management systems. They are another type of system that is often confused with log management, thanks to the somewhat unclear definition of each. And finally, we’ll review the best log management for Linux.

What Is Log Management?

Before we can talk about log management, let’s define what a log is. Simply defined, a log is the automatically-produced and time-stamped documentation of an event relevant to a particular system. In other words, whenever an event takes place on a system, a log is generated. Systems and devices will generate logs for different types of events and many systems give administrators some degree of control over which event generates a log and which doesn’t.

As for log management, It is simply referring to the processes and policies used to administer and facilitate the generation, transmission, analysis, storage, archiving and eventual disposal of large volumes of log data. Although not clearly stated, log management implies a centralized system where logs from multiple sources are collected. Log management is not just log collection, though. It is the management part which is the most important. And log management systems often have multiple functionalities, collecting logs being just one of them.

Once logs are received by the log management system, they need to be standardized into a common format as different systems format logs differently and include different data. Some start a log with the date and time, some start it with an event number. Some only include an event ID while others include a full-text description of the event. One of the purposes of log management systems is to ensure that all collected log entries are stored in a uniform format. This will event correlation and eventual searching much easier down the line.

Even correlation and searching are two additional major functions of several log management systems. The best of them feature a powerful search engine that allows administrators to zero-in on precisely what they need. Correlation functions will automatically group related events, even if they are from different sources. How—and how successfully—different log management system accomplish that is a major differentiating factor.

ALSO READ: 15 Best Network Monitoring Tools (Our Own Review)

Logging Technologies

Log management would be much more difficult, perhaps not even possible, if it were not for logging protocols. A few of them exist. They define what data is to be included in logs, how that should be formatted and, sometimes, how they are to be transmitted between systems.

Syslog is arguably the most-used logging protocol, especially in the Linux world. The technology was invented in the early eighties and has become the de-facto standard for all Unix-like systems. One of the greatest assets of the syslog technology is how it facilitates the separation between the system or software that generates logs, the system that stores them, and the software that reports and analyzes them. Using the Syslog technology makes log management much easier. And Syslog is not a Unix exclusive. Many non-Unix devices such as switches, routers and all sorts of equipment from many vendors use a variant of the syslog protocol.

There are other logging technologies. Microsoft Windows, for example, uses a different logging system. It might have to do with the fact that Windows operating systems and applications have logs that typically contain more detailed information than the Syslog technology permits. Fortunately, the Windows Event Collector functions provide a mean for log management that various systems can use to receive events from Windows hosts. This post is about Linux log management so let’s not waste too much time on Windows, though.

No matter what logging technology is used, an important part of log management is configuring devices to send their logs to the management system. Other types of tools such as network monitoring systems can fetch data from the systems they monitor but with log management, each device must be “told” where to send its logs. It is, however, a relatively simple task which is often accomplished by issuing a simple command.

FURTHER READING: Best Network Diagram Mapping and Topology Software

Log Servers or Log Management?

Since it has been available on every Unix-like system—including Linux—for a quite a while, Syslog is often used as a log server with one computer receiving Syslog data from several others. While this centralized storage of logs has definite advantages, it is not enough to be called log management.

To deserve the Log Management System name, a product must include at least some of the more advanced functions. According to Wikipedia, “log management is comprised of the following functions: log collection, centralized log aggregation, long-term log storage and retention, log rotation, log analysis, log search, and reporting”. Wow! That’s a lot of functionality. Log servers, on the other hand, often only offer the log collection and storage and rarely more than that.

A Word (Or Two) About SIEM

Another popular technology that is associated with logs and often confused with log management systems is Security Information and Event Management, or SIEM. This is different from log management yet it is closely related. The line is so thin between them that some products advertised as log management systems are actually SIEM systems while some basic SIEM systems are nothing more than advanced log management systems.

The confusion stems from the fact that log management—or, at the very least, log analysis—is an important component of SIEM systems. What differentiates SIEM systems is that they perform log analysis with the ultimate goal of identifying security issues. They will, for instance, look for signs of unsuccessful logins which could be a tell-tale sign of an unauthorized intrusion attempt. These systems continuously scan log entries looking for anything out of the ordinary. While some SIEM systems do include extensive log management features, some use an external log management system and it’s not uncommon to see both systems running side by side.

RELATED READING: Best IP Scanners for Mac

The Best Log Management For Linux

Hopefully, we now have a common understanding of what log management is and what it isn’t. So, let’s have a look at what’s available for Linux. But first, let’s clarify something. When referring to Linux log management, what we mean are log management systems that can accommodate Linux logs and that will either run on the Linux platform or in the cloud. Some of our selections—particularly cloud-based systems—will also work with logs from other platforms.

1. SolarWinds Papertrail (FREE PLAN AVAILABLE)

SolarWinds has become a household name among network administrators. It’s making some of the best tools for almost 20 years, bringing us great bandwidth monitoring tools and one of the best NetFlow analyzers and collectors. The company is also well-known for publishing several free tools that address some specific needs of network administrators such as subnet calculator or a syslog server.

SolarWinds Papertrail Dashboard

Not so long ago, SolarWinds acquired Papertrail, a popular log management system. It aggregates log files from a wide variety of popular products like Apache or MySQL as well as Ruby on Rails apps, different cloud hosting services and other standard syslog and text-based log files. Papertrail users can then use the web-based search interface or command-line tools to search through these files to help diagnose various issues. Papertrail also integrates with other SolarWinds products such as Librato and Geckoboard for graphing results.

Papertrail is a cloud-based, software as a service (SaaS) offering from SolarWinds. Being cloud-based means that it will work fine in an all-Linux environment. The platform is easy to implement, use, and understand, and it will give you instant visibility across all systems within minutes. Furthermore, the product has a very effective search engine that can search both stored and streaming logs. And it is lightning fast.

Papertrail is available under several plans including a free plan. It is somewhat limited, though, and only allows 100 MB of logs each month. It will, however, allow 16 GB of logs in the first month which is equivalent to giving you a free 30-day trial. Paid plans start at $7/month for 1GB/month of logs, 1 year of archive and 1 week of index. Noise filtering allows the tool to preserve data by not saving useless logs.

2. Loggly

Loggly is another cloud-based online service. Primarily a log consolidator, it also offers log analysis functionality. As a virtue of being cloud-based, this system requires no installation and is ready to use the minute you subscribe. Of course, your systems and devices will need to be configured to upload their standard log files periodically to the online server.

Loggly Screenshot

Loggly then converts the received log data into a standard format, thereby allowing the analyzer to process records from various sources and enabling events tracking and correlation across all systems, regardless of their operating system or logging technology. The sources of log data are not limited to your on-premises servers. The system is, of course, able to process logs generated by online servers, such as Amazon’s AWS and it can include messages created by specific applications such as Docker and Logstash, just to name a few.

The Loggly service is available under three different plans, with increasing data processing limits and retention times. You need to pick the right one to give you enough space for your log data. The entry-level plan is called Loggly Lite. It is free to use. Under this plan, you can upload 200 MB of log data per day and the system will retain each record for seven days. Next is the Standard plan which gives you an upload allowance of 1 GB per day and retains records for 30 days. Paid plans also let you use multiple user accounts. With the Standard package, you can have three user accounts. The top tier is called Loggly Enterprise. It has no limit to the number of users accounts you can set up and prices vary depending on the amount of upload capacity and the retention period that you require. Payment for all paid plans can be either monthly or annually and a free 14-day trial is available on the Standard plan.

3. Splunk

Splunk is a well-known—within the system administration community—comprehensive log management system for Linux, Mac OS, and Windows. More than just a basic log management system, some consider it to be a full-fledged intrusion prevention system. The product is available in three versions. At the top is Splunk Enterprise which is more of a network management system rather than just a log management tool. Pricing starts at $173 per month and you get a lot of functionality.

Splunk Log Management Screenshot

There is also a free version of Splunk which is basically the same tool without some of its most advanced functionalities. In essence, it is restricted to log file analysis. You can feed in any of your standard logs files or send it live data through a file into the analyzer. The free version has a few limitations. It can, for instance, only have one user account and its data throughput is limited to 500 MB of logs per day. Data sorting and filtering functionality is built into Splunk, facilitating your troubleshooting efforts. You can use these features for dividing log records by date and writing each group out to new files. In fact, this functionality is very flexible.

4. Nagios Log Server

Nagios is best known for its excellent network monitoring software but its Log Server is just as interesting. The product is simply called the Nagios Log Server and it offers centralized log management, monitoring, and analysis. This tool can greatly simplify the process of searching your log data. It also lets you set alerts to be notified of potential threats Furthermore, the software has high availability and fail-over built right into it. Furthermore, its easy source setup wizards will help you quickly configure servers to send all log data and start monitoring your logs in minutes.

Nagios Log Server Real-Time Data

The Nagios Log Server allows for an easy correlation of log events across all servers in just a few clicks. The system will let you view log data in real-time, giving you the ability to analyze and solve problems as they occur. The product features impressive scalability and it will continue to meet your needs as your organization grows. Additional Nagios Log Server instances can be added to a monitoring cluster, allowing you to quickly add more power, speed, storage, and reliability.

The single-instance price for the Nagios Log Server is $3 995 and although a free trial doesn’t appear to be available, a free online demo is, should you prefer to have a first-hand look at the product.

5. Graylog

Next on our list is a product called Graylog. The product offers many interesting features. The tool will parse and enrich logs and event data from any data source. Its processing pipelines allow for some flexibility in routing, blacklisting, modifying and enriching messages in real-time. Graylog will search through terabytes of log data to discover and analyze important information. The powerful search syntax lets you find exactly what you are looking for.

Graylog Screenshot

With Graylog, you can create dashboards to visualize metrics and observe trends in one central location. You can use field statistics, quick values, and charts from the search results page to dive in for deeper analysis of your data. The system also has the option to trigger actions or issue notifications on events such as failed login attempts, exceptions or performance degradation.

Graylog is a free, open-source log file-based system that can give you a lot more functionality than just a log archiving utility. This log analyzer has a graphical user interface and it can run on Ubuntu, Debian, CentOS, and SUSE Linux. You can also run it on a virtual machine on Microsoft Windows and you can install the Graylog system on Amazon AWS.

6. ManageEngine EventLog Analyzer

ManageEngine, another common name among network administrator, makes an excellent log management system called the ManageEngine EventLog Analyzer. The product will collect, manage, analyze, correlate, and search through the log data of over 700 sources using a combination of agentless and agent-based log collection as well as log import.

ManageEngine EventLog Analyzer

Speed is one of the ManageEngine EventLog Analyzer’s strength. It can processes log data at an impressive 25,000 logs/second and detect attacks in real-time. It can also perform fast forensic analysis to reduce the impact of a breach. The system’s auditing capabilities extend to the network perimeter devices’ logs, user activities, server account changes, user accesses, and more, helping you meet security auditing needs.

The ManageEngine EventLog Analyzer is available in a feature-reduced free edition which only supports 5 log sources or in a premium edition which starts at $595 and varies according to the number of devices and applications. A free, full-featured 30-day trial version is also available.

Read 6 Best Log Management Tools For Linux in 2019 by Renaud Larue-Langlois on AddictiveTips – Tech tips to make you smarter

5 Best Network Traffic Analyzers (2019 Review)

Network Traffic Analyzers let network administrators and managers get an excellent grasp of not only how much a network is utilized but, more importantly, HOW it is utilized. It is one thing to know that a given network segment suffers from congestion but being able to figure what is causing that congestion gives you a whole new perspective. Without that information, the only option for fixing congestion issues is to throw more bandwidth at it—a temporary solution at best. Bandwidth being still expensive, there are certainly better ways to address this type of issue.

Network traffic analysis may hold the answer and today, we’ll explain what it is and review some of the best tools you can use.

Network traffic analyzers

We’ll begin our journey into network traffic analysis with some useful theory. We’ll first have a closer look at what it actually is. This is important as we want everyone to be on the same page for the remainder of our discussion. Next, we’ll introduce NetFlow and other similar flow-reporting systems and protocols. They are at the core of most traffic pattern analysis tools. Without them, there possibly wouldn’t be any network traffic analysis.

We’ll start by having a look at Cisco’s NetFlow technology and its multiple variants before we have a look at S-Flow, a competing system that is somewhat different in how it operates although it serves a similar purpose. With all this information, we’ll be ready to review the top network traffic analyzers that are currently available.

About Network Traffic Analysis

In its simplest expression, network traffic analysis—sometimes called pattern analysis—is the process of recording, reviewing and/or analyzing network traffic for the purpose of performance, security and/or general network operations management. More specifically, it is the process of using manual and automated techniques to review granular-level details and statistics about ongoing network traffic.

There are primarily two types of network traffic monitoring. The first is bandwidth utilization monitoring which can provide quantitative data. This type of monitoring will let you see how much traffic is going by at a specific point on a network but it won’t provide any data on the nature of this traffic. The second type of monitoring, the one that we’re discussing today and which is referred to as network traffic analysis goes deeper and its primary objective is to offer an in-depth insight into what type of traffic, network packets or data is flowing through a network as well as its source and destination.

Although network traffic analysis can be done manually, it is would be a rather tedious endeavour and it is most often done using network monitoring tools. The traffic statistics obtained from network traffic analysis can help with understanding and evaluating the network’s utilization. It will reveal important data on the type, size, origin, and destination of data packets. It can even include some information on the content of data packets.

Network security teams can use network traffic pattern analysis to identify malicious or suspicious packets within the traffic. Likewise, network administrations seeking to monitor download and upload speeds, throughput, content, etc. will use it to better understand network usage.

On the downside, network traffic pattern analysis can also be used by attackers and/or intruders to analyze network traffic patterns and identify vulnerabilities or means to break in or retrieve sensitive data. This, like many technologies, is a double-edged sword.

RELATED READING: 8 Best Network Latency Testing Tools (2019 Reviews)

NetFlow And Other Flow-Reporting Systems

NetFlow is a feature that was introduced on Cisco routers back in the mid-1990s, give or take a year or two. The technology offers the possibility to collect IP network traffic as it enters or exits an interface. This is different from bandwidth monitoring where data is counted but not collected. By analyzing the collected data, one can determine things such as the source and destination of traffic, class, and type of service, and, ultimately use this information to identify the causes of congestion or other network issues.

A typical NetFlow monitoring setup is made up of three main components:

  • The flow exporter aggregates packets into flows and exports flow records towards one or more flow collectors. This is the component that resides within the networking device.
  • The flow collector is responsible for the reception, storage and pre-processing of flow data received from a flow exporter.
  • The flow analyzer analyzes the received flow data in the context of intrusion detection or traffic profiling, for example.

A flow, in NetFlow parlance, is a unidirectional sequence of packets that share a certain number of attributes such as their ingress interface, source and destination IP addresses, IP Protocol (TCP/UDP/ICMP, etc.), source and destination IP ports, and IP type of service. In other words, it corresponds to a network session. Detailed data about each individual flow is collected by the flow exporter before being exported to the flow collector. In most instances, the flow collector and analyzer are two components of the same system and we rarely see them separated.

ALSO READ: 5 Best Tools for Traffic Pattern Analysis

Once a Cisco-exclusive, NetFlow is now available on equipment from many vendors including Juniper, Alcatel-Lucent, and Nortel, just to name a few. Some vendors call it a different name such as J-flow for Juniper. There’s even a relatively recent IETF-standardized version called IPFIX which stands for Internet Protocol Flow Information eXport.

There is also sFlow from InMon, a somewhat equivalent yet widely different technology. sFlow uses similar methods for collecting flow information but adds data sampling—hence the S—for even more detailed information. Only a few NetFlow analyzers and collectors can handle sFlow data as the two are too different.

The Best Network Traffic Analyzers

There are way too many network traffic analyzers using NetFlow or sFlow, potentially making the selection process a daunting challenge. To help you, we’ve put together this list of some of the very best tools for network traffic analysis. Each of them is worth giving it a look.

1. SolarWinds NetFlow Traffic Analyzer (FREE TRIAL)

First on our list is the SolarWinds NetFlow Traffic Analyzer or NTA. If you don’t know SolarWinds, the company has acquired a top reputation for making some of the best network management tools. Its flagship product, the Network Performance Monitor is one of the best bandwidth monitoring tools available. The company is also known for its great free tool addressing specific network administration needs such as one of the best subnet calculators or TFTP servers.

As its name implies, the SolarWinds NetFlow Traffic Analyzer uses the NetFlow protocol to collect detailed information on what the observed traffic is. It can, for instance, report on what type of traffic is more frequent or what user or device is using the most bandwidth. Several different views are available on the tool’s dashboard such as top applications, top protocols or top talkers, for instance. The tool will support most NetFlow variants from different manufacturers.

SolarWinds NTA Dashboard Summary

Among some of the SolarWinds NetFlow Traffic Analyzer’s best features:

  • It can be used to can monitor network usage by application, protocol, and IP address group.
  • It will monitor Cisco NetFlow, Juniper J-Flow, sFlow, Huawei NetStream, and IPFIX flow data to identify which applications and protocols are the top bandwidth consumers.
  • It will collect traffic data, correlate it into a usable format, and present it on its web-based user interface
  • It can help you identify which applications and categories consume the most bandwidth for better network traffic visibility and it has support for Cisco NBAR2.

The SolarWinds NetFlow Traffic Analyzer is available as an add-on to the Network Performance Monitor (NPM). Prices start at $1,915 for 100 nodes. The number of nodes you purchase must match your NPM license. If you don’t already own the NPM software, that will cost $2,995 for the same 100 nodes level. And if you want to try it before you buy it, you can download a fully functional 30-days evaluation version of either or both products,

2. PRTG Network Monitor

The PRTG Network Monitor, or simply PRTG from Paessler AG, is an all-in-one solution whose primary purpose is monitoring bandwidth utilization. As such it integrates SNMP bandwidth monitoring and NetFlow collection and analysis. But it doesn’t stop there and PRTG uses what they call sensors to monitor various systems, devices, traffic, and applications. Here’s a rundown of some of the most important monitoring technologies supported:

  • Flows (like NetFlow or sFlow)
  • SNMP with ready to use and custom options
  • WMI and Windows Performance Counters
  • SSH for Linux/Unix and macOS systems
  • Packet Sniffing
  • Ping, SQL and many more

PRTG NetFlow Sensor Screenshot

Installing PRTG is easy. In fact, Paessler claims you could be done within a couple of minutes. After running the installer, the auto-discovery process will discover devices and set up basic sensors. You can then add more advanced and complex sensors—such as NetFlow collectors—manually. Should you need it, a detailed video will show you how it’s done.

PRTG runs on Windows but its user interface is web-based and can be accessed from any browser on any platform. There are also mobile apps for Android and iOS. Talking about the mobile apps, this product has a unique feature in the form of QR code labels that you can print and affix on your devices. It is then a simple matter of scanning the code from the mobile apps to quickly view the device’s sensor data.

The PRTG network monitor is available in two versions. There’s a free version that is limited to 100 sensors. Each and every monitored element counts as one sensor. For example, to monitor each port of a 48-port switch, you’ll need 48 sensors. For NetFlow collection and analysis, you’ll need one sensor for each flow exporter. For more than 100 sensors, you need a paid license. They’re available for 500, 1000, 2500, 5000, and unlimited nodes at prices varying from around $1 600 to just under $15 000. Note that the free version will allow for unlimited sensors for the first 30 days giving you a chance to thoroughly test-drive the product.

3. Scrutinizer

Scrutinizer from Plixer is an excellent NetFlow Analyzer. It’s actually much more than that many see it as a full-fledged incident response system. It has the ability to monitor different flow types such as NetFlow, J-flow, NetStream, and IPFIX, so you’re not limited to monitoring only Cisco devices.

Scrutinizer NetFlow Analyzer Screenshot

Scrutinizer features a hierarchical design and offers a streamlined and efficient data collection. This lets one start small and easily scale way up to millions of flows per second. Scrutinizer claims to help you quickly find the real root cause of most network issues. The product can work in both physical and virtual environments and it comes with advanced reporting features.

Scrutinizer is available in four license tiers from the basic free version to the top-tier SCR level which can scale up to over ten million flows per second. The free version is limited to ten thousand flows per second and it will only keep raw flow data for 5 hours. In-between tiers are the MDX level which keeps data for 25 hours and the SSRV which keeps it forever. You can try any license tier for 30 days after which it will revert back to the free version.

4. ManageEngine NetFlow Analyzer

ManageEngine is another popular name among network administrators. The company makes some excellent tools paid as well as a few free ones. The ManageEngine NetFlow Analyzer provides a detailed view of a network’s bandwidth utilization as well as traffic patterns. It supports most flow technologies including NetFlow, IPFIX, J-flow, NetStream and a few others. The platform also boasts a web-based user interface which offers an impressive number of different views on your network. It will, for instance, let you view traffic by application, by conversation, by protocol, and several more viewing options. You can also set alerts to warn you of potential issues. You could, for instance, set a traffic threshold on a specific interface and be alerted whenever traffic exceeds it.

ManageEngine NetFlow Analyzer

Much of the ManageEngine NetFlow Analyzer’s strength comes from its impressive reports. The product has several useful pre-built reports that are tailored for specific purposes such as troubleshooting, capacity planning or billing. If you’d rather have customized reports, they can easily be created.

The product’s dashboard is just as impressive as its reports. It includes several pie charts depicting top applications, top protocols or top conversations, for example. It can also display a heat map showing the status of the monitored interfaces. Dashboards can be customized to include only the information you need. For the on-the-go admins, there’s a smartphone app that will let you access the dashboard and reports from wherever you are.

The ManageEngine NetFlow Analyzer comes in two versions. The free version limits you to monitoring only two interfaces or flow exporters. For greater capacity, licenses are available in several sizes from 100 to 2500 interfaces or flows at prices varying between about $600 to over $50K plus annual maintenance fees. A free 30-day trial is available on all paid plans.

5. sFlowTrend

As we explained, NetFlow and sFlow protocols are quite different and it is rare for one tool to support both. Among all the products reviewed so far, only the PRTG Network Monitor supports the sFlow protocol. But if your network is primarily made of sFlow-enabled devices, here’s one of the best tools we could find.

sFlowTrend is an sFlow monitoring tool from inMon, the company behind the sFlow protocol. It is a basic and somewhat limited yet very capable tool. There’s a free version that will let you gather data from up to five sFlow-enabled devices and will only keep history data in RAM for up to an hour. While this could be enough to troubleshoot some networking issues, it’s not what you need for ongoing monitoring. For a more complete tool, you need to upgrade to the paid pro version which removes the number of devices limit and stores history data to disk.

sFlowTrend V6

The sFlowTrend dashboard provides a quick view of the current state of your network and its components. It will display top-level thresholds and interfaces with potential errors. Clicking on the sFLowTrend Network tab reveals summarized performance statistics and detailed traffic at the network or device level. Alerting thresholds can be used to receive alerts when higher-than-usual bandwidth usage is observed or network errors happen. The software also features a Root Cause tab where you can drill down on the cause of an issue such as a threshold violation.

The sFlowTrend Hosts tab is where you’ll find more detailed information about each device. It can display performance data on CPU, disk, and more, for sFlow-enabled servers. The Services tab is where you’ll find performance data for applications that export sFlow data. And on the Events tab, you’ll find a log of events such as exceeded thresholds or detected errors. Finally, the Reports tab offers several predefined reports and also supports the creation of custom reports.

sFlowTrend is written in Java and comes with both a Java-based or plain web-based user interface. It is available for Windows, Mac, and Linux. The software features an excellent online help system to assist you in configuring and using the tool.

Wrapping Up

No matter which tool you choose, network traffic analyzers will give you an invaluable insight into what goes on in your network. The tools we’ve reviewed each provide excellent value and picking one will most likely be a matter of personal preference as there might be a specific feature in one of the tools that particularly appeals to you. With all the paid tools offering either a free trial or a free version, there’s no reason why you couldn’t try a few before making a decision.

Read 5 Best Network Traffic Analyzers (2019 Review) by Renaud Larue-Langlois on AddictiveTips – Tech tips to make you smarter

Best Real-Time Bandwidth Monitoring Utilities to Track Network Usage

If only networks had infinite bandwidth, wouldn’t life be easier? Unfortunately, they don’t. Network congestion is still one of the biggest problems of every network. It happens when the actual bandwidth approaches or exceeds what’s available. As a rule of thumb, network administrators try to keep bandwidth utilization below 70% of the available bandwidth. This means that, on a 1 Gb/s interface, there should never be more than 700 Mb/s of actual traffic. The best way to keep that from happening is by keeping a close eye on the actual network traffic.

While several tools can be used for that purpose, most only provide average utilization figures over a period of time. Today, we’re having a look at real-time bandwidth monitoring.

Best Real-Time Bandwidth Monitoring Utilities to Track Network Usage

We’ll start off by describing the different methods that can be used to monitor bandwidth, As you shall soon see, there are mainly three of them, and one of them won’t give you true real-time figures although one can cheat and get near real-time data. More about that in a moment. Then, we’ll have a look at some of the best tools you can use for real-time bandwidth monitoring. Our list has tools of all three types.

Monitoring Bandwidth

Congestion is the number one enemy of networks. We all know that. Think of a network as a highway where congestion is similar to traffic jams. However, unlike automobile traffic which you can easily see, network traffic happens within cables, switches and routers or over the air where it remains invisible. This is where network bandwidth monitoring tools can come in handy. They give network administrators the visibility they need to make sure things keep running smoothly.

Monitoring bandwidth in real-time is particularly interesting, especially when troubleshooting performance issues. Many monitoring tools use technologies that poll devices at rather long intervals and compute average utilization figures. Consequently, it is easy to miss short bursts of high utilization that will often be averaged out b traditional tools. Talking about the tools, let’s have a look at what they are.

ALSO READ: Top 10 Intrusion Detection Tools

Network Monitoring Tools

There are several ways that can be used to monitor network bandwidth utilization. The first is to capture packets at a given point on the network. You can also use SNMP to poll devices for interface statistics and finally, you can have devices that support the technology send out NetFlow of S-Flow information.

Packet Capture

Packet capture used to be the number one way of analyzing network traffic. For a while, it was the only way. And it is still used a lot to pinpoint specific network issues but it’s rarely used for bandwidth monitoring anymore. With packet capture, every data packet in and/or out of a specific device’s interface is captured and decoded. This can easily give you a real-time view of what’s going on. However, bandwidth monitoring has no real use for the content of each packet but only its size. Although very efficient, many administrators and engineers feel that using packet capture for real-time bandwidth monitoring is a gross overkill.

FURTHER READING: Best Network Diagram Mapping and Topology Software

SNMP

The Simple Network Management Protocol—or SNMP—is a huge and very complex protocol that can be used to remotely monitor, configure and modify different types of networking equipment. The only thing simple about it is its name, though. Implementing it can be a rather complex task. Using this technology, SNMP-enabled devices make a certain number of parameters available.

When it comes to bandwidth monitoring, two of those parameters are of interest. They are called bytes in and bytes out and they are available for each network interface. By reading these values periodically, you can compute the number of bytes per unit of time which is exactly what bandwidth is.

The precision you get depends on the polling interval. SNMP monitoring tools typically poll devices every 5 minutes, thereby computing 5-minute average utilization. However, by using a very short polling interval (one second, for example) one can get near real-time measurements.

Flow Analysis

Originally developed by Cisco Systems, NetFlow is, as its name suggests, a network flow analysis system. Devices that support NetFlow—or one of its multiple cousins such as J-flow or IPFIX—collect information about each data flow—hence the name—which they then sent to a NetFlow collector and analyzer. This flow information contains quantitative information about the flow and, since it is sent as soon as a flow terminates, you get near real-time data.

RELATED READING: Best IP Scanners for Mac

The Best Tools For Real-Time Bandwidth Monitoring

Our picks of the best tool include tools in all three categories. Some are specifically designed for real-time bandwidth monitoring while others can be configured to provide that type of metric. Others yet will provide usage data from which real-time bandwidth usage can be extrapolated. Let’s have a look at what they are.

1. SolarWinds Real-Time Bandwidth Monitor (FREE DOWNLOAD)

Every network administrator should know SolarWinds. The US-based company has been making great network administration tools for about 20 years. It benefits from an excellent reputation for making some great free tools that are distributed in a no-strings-attached fashion. SolarWinds also makes several commercial tools that are among the best on the market. Its flagship product, the Network Performance Monitor (NPM) is a complete monitoring solution that can be scaled to networks of any size, from the smallest to the largest.

One of SolarWinds’ best free tools—and our number one pick—is the SolarWinds Real-Time Bandwidth Monitor. The software, which runs on Microsoft Windows, uses SNMP to poll multiple network devices and get traffic statistics from their various interfaces. The results are shown visually on graphs depicting each interface’s usage statistics.

SolarWinds Real-Time Bandwidth Monitor

Configuring the tool is a simple matter of providing a device’s IP address or hostname and SNMP parameters such version and community string. The real-time bandwidth monitor then displays a list of available interfaces on the chosen device also with some basic data about them. Selecting a specific interface by clicking it reveals a graphic displaying in real-time the inbound and outbound bandwidth usage on the selected interface. You can also set alert thresholds to be notified whenever usage exceeds a predefined limit on any interface.

There are some limitations to this free tool. For instance, only one device can be monitored at a time. Also, usage history is only kept for 60 minutes. This makes the tool a great asset for troubleshooting purposes but probably not for long-term utilization evolution surveying. For a more comprehensive package, the SolarWinds Bandwidth Analyzer Pack can be purchased.

The SolarWinds Real-Time Bandwidth Monitor is distributed as a bundle along with the SolarWinds Network Analyzer, another great free tool that you can use to monitor NetFlow-enabled devices. That tool will let you drill down by conversation, application, endpoints or protocol which the real-time bandwidth monitor won’t.

You can get the bundle with both the free Real-time Bandwidth Monitor AND the Network Analyzer by visiting SolarWinds’ website.

2. SolarWinds Deep Packet Inspection and Analysis Tool (FREE TRIAL)

Another excellent tool from SolarWinds that can help you with real-time bandwidth monitoring is the Deep Packet Inspection and Analysis Tool. It comes as a component of SolarWinds’ flagship product, the Network Performance Monitor. It is a type of packet capture tool yet its operation is quite different from more “traditional” packet sniffers.

SolarWinds Deep Packet Inspection and Analysis

To summarize the tool’s functionality: it will help you find and resolve the cause of network latencies, identify impacted applications, and determine if slowness is caused by the network or an application. The software will also use deep packet inspection techniques to calculate response time for over twelve hundred applications. It will also classify network traffic by category, business vs. social, and risk level, helping you identify non-business traffic that may need to be filtered or otherwise eliminated.

The SolarWinds Deep Packet Inspection and Analysis Tool comes as part of the Network Performance Monitor which, in itself, is an impressive piece of software with so many components that a whole article could be dedicated to it. At its core, it is a complete network monitoring solution that combines the best technologies such as SNMP and deep packet inspection to provide as much information about the state of your network as possible. The tool, which is reasonably priced comes with a 30-day free trial so you can make sure it really fits your needs before committing to purchasing it.

3. SolarWinds NetFlow Traffic Analyzer (FREE TRIAL)

The SolarWinds NetFlow Traffic Analyzer is simply one of the best flow analysis tools there are. It installs on top of the SolarWinds Network Performance Monitor, another great tool, and adds a unique set of information about your network’s traffic. You can use the tool for monitoring bandwidth usage by application, by protocol, and by IP address group. It will monitor Cisco’s NetFlow, IPFIX, Juniper’s J-Flow, sFlow, and Huawei’s NetStream flow data allowing it to identify which devices, applications, and protocols are the highest bandwidth consumers. It will also collect traffic data, correlating it into a usable format, and presenting it to the user through a web-based interface for monitoring network traffic. Furthermore, it can identify which applications and categories consume the most bandwidth for better network traffic visibility.

SolarWinds NetFlow Traffic Analyzer Dashboard

The SolarWinds NetFlow Traffic Analyzer is an add-on module to the SolarWinds Network Performance Monitor. It uses the NPM’s database and node management facilities. As such, the most important requirement is that you have the Network Performance Monitor installed and configured first.

The SolarWinds NetFlow Traffic Analyzer is licensed based on the underlying Network Performance Monitor license. If you already own the Network Performance Monitor, you need to make sure the license level selection is the same for both products. If you do not already own NPM, keep reading for more options. Like NPM, NTA is available in five licensing tiers, based on the number of monitored nodes. Prices start at $1 945 for 100 nodes. If you don’t already have a SolarWinds Network Performance Monitor license, both products can be purchased together as the Network Bandwidth Analyzer Pack. If you want to give these two great products a test run and see first-hand what they can do for you, a free 30-day trial version is available from SolarWinds.

4. ManageEngine SNMP Bandwidth Monitoring Software

ManageEngine is self-described as a company that “has complete and easy solutions for even your most difficult IT management problems, from keeping your business safe to ensuring high availability to making your users happy.” This is a bold statement but it describes the company quite well. ManageEngine is well-known for its high-quality tools including several tools aimed at monitoring different aspects of networks.

And just like SolarWinds, ManageEngine is also famous for its free tools. Of particular interest in the context of this article is the SNMP Bandwidth Monitoring Software. It is offered as part of ManageEngine’s free OpUtils bundle, a huge pack of some 16 network management utilities. It runs on both Windows and Linux and the free edition allows monitoring up to 10 devices and their interfaces.

ManageEngine SNMP Bandwidth Monitor

Setting the tool up, just like it’s almost always the case, requires several steps. You first specify a subnet to scan and some SNMP parameters to use. The tool will then discover devices on the specified subnet. Once the devices are discovered, you can view their interface’s statuses from the inventory tab. You can also display graphs of network speed and bandwidth usage.

For each interface, you can generate reports of bandwidth usage over the past 12 hours to one month. Furthermore, you can set alert thresholds and be notified by email or SMS text messages whenever they are reached.

The ManageEngine SNMP Bandwidth Monitoring Software is ideal if your network is small with no more than 10 devices. If you manage a bigger network, ManageEngine also has paid version with no device limitation that you may want to try. To make it easier, ManageEngine offers a free 30-day evaluation version of its full OpsUtil software. In fact, the free version is first installed as a 30-day trial and revert to limited features after the trial period ends.

5. PRTG Network Monitor

According to Paessler, its publisher, you can set up the PRTG Network Monitor and be up and running in a couple of minutes. Our experience shows that it might take you a bit longer than that to get it completely configured to your liking and monitoring all your devices but we have to admit that setting the product up was an exceptionally easy experience.

PRTG Device Overview

Feature-wise, PRTG is an impressive product. For starters, the product comes with several different user interfaces. There’s a native Windows enterprise console, an Ajax-based web interface as well as mobile apps for Android and iOS. And the different interface makes full use of each device’s capabilities. For example, PRTG allows you to print QR code labels that you can affix to your devices. Then, scanning the code from the mobile app will quickly take you to the device’s graphs.

And talking about graphs, PRTG leaves nothing to be desired. It can not only monitor and graph bandwidth utilization but also many more parameters using SNMP, WMI, NetFlow, and sFlow. It also has some amazing reports that can be viewed as HTML or PDF or exported to CSV or XML to be processed externally. The reports can be run on-demand or be scheduled to run automatically.

The Paessler website lets you download two different versions of PRTG. You can choose either the free version of the free 30-day trial version. The former will limit you to monitoring up to 100 sensors. In PRTG parlance, a sensor is each parameter that you want to monitor. For example, monitoring bandwidth on each port of a 48-port switch will require 48 sensors. And if you also want to monitor the switch’s CPU and memory loads, you’ll need two more sensors. As you can see, they can quickly add up.

6. Wireshark

Wireshark is the reference in packet sniffers. It has become the de-facto standard and most other tools tend to emulate it. This tool will not only capture traffic, but it also has quite powerful analysis capabilities. So powerful that many administrators will use tcpdump or Windump to capture traffic to a file then load the file into Wireshark for analysis. This is such a common way of using Wireshark that upon startup, you’re prompted to either open an existing pcap file or start capturing traffic. Another strength of Wireshark is all the filters it incorporates which allow you to zero in on precisely the data you’re interested in.

Wireshark Screenshot

To be perfectly honest, Wireshark tends to have a steep learning curve but it is well worth learning. It will prove invaluable time and time again. And once you’ve learned it, you’ll be able to use it everywhere as it has been ported to almost every operating system and it is free and open-source.

Read Best Real-Time Bandwidth Monitoring Utilities to Track Network Usage by Renaud Larue-Langlois on AddictiveTips – Tech tips to make you smarter

Free website visitors geolocation with the ipstack API (Review)

Since its humble beginnings over 25 years ago, the world wide web has evolved in ways that even the best analysts could have never predicted. Form what was essentially a tool to help researchers share information, it has evolved into a consumer tool that many of us simply can’t live without. It has changed many aspects of our lives and we’ve come to rely on it, perhaps more than we should. It has become so commonplace that it is changing the retail industry. From big chains of brick-and-mortar stores, the top retailers are now web-based marketplaces.

To be successful, online retailers need not only to be present on the web, but they also need to fully utilize its features. One such feature is geolocation. This technology can allow a retailer to know the location of its website’s visitor, a useful feature to allow them to display estimated shipping charges or to present location-specific offers. One of the easiest ways to benefit from geolocation is to use a dedicated API that can be fed with an IP address and that will return any pertinent information about that address. The ipstack API is one such product and we’re about to discover how it does its magic and what it can do for you. We’ll have a look at the tool’s basic functionality as well as how it to use it. We’ll see how it performs, what type of documentation and support is available for the product and have a look at its rather simple pricing structure.

Introducing ipstack

At its core, ipstack is a geolocation API which uses multiple interconnected ISPs to provide accurate geographical data. You call the API from your website’s code and it will return a plethora of geographical data about your website’s users. You will, of course, get the user’s geographical location (isn’t that the main point?) but that’s not all. For instance, the tool will return a link to an image of the user’s country flag or an emoji version of it. Furthermore, you’ll get information about whether the user is accessing your site through a proxy. We’ll go in deeper details on all this as we look in deeper details at the available modules.

The project, initially called freegeoip.net, saw the light back in 2009 as an open source initiative to help developers. Through continuous improvement based on user feedback, it evolved some eight years later into the full-fledged product that we are looking at. During this time, the product’s user base grew up to over one hundred thousand satisfied customers. And they’re not just any customers. Giants like Microsoft or Samsung are using the API. So are Airbnb and HubSpot.

The ipstack API uses a highly dynamic scheme which updates its database multiple times a day, ensuring the integrity of the returned results. For this purpose, ipstack has partnered up with several large ISPs throughout the world. The tool performs all communication over 256-bit SSL-encrypted channel—using HTTPS—to secure all data transfers. Results can be returned by the API in either XML or JSON format, depending on your specific needs.

One of the best parts of ipstack API is its modularity. Various available module return increasingly more data about the user’s location. That makes for a very flexible tool. Speed is also one of the product’s forte. You call the API with the appropriate parameter(s) and get the requested results super fast. Currently, ipstack API handles more than two billion requests daily and it’s still responding almost instantly. Through a partnership with various ISPs, the product can cover some 2 million locations and over two hundred thousand cities worldwide. It also supports both IPv4 and IPv6 addresses.

How Geolocation Works

Before we go any further, let’s briefly pause and have a quick look at how geolocation looks. As you’d guess from ipstack’s name, it has to do with IP addresses. IP addresses can be compared—to a certain extent—to postal addresses. Each Internet-connected device has a unique IP address. But these IP addresses are not just randomly assigned to every device. There is some sense into it. Part of an IP address identifies the individual connected device and part of it identifies the network it is connected to. And the network part itself is hierarchically organized, somewhat like a computer disk directory tree. (I hope the engineers among our readers will forgive me for this oversimplification.)

This is similar to a postal address with a street number, a street name, a city, and, eventually, a country. And just like the British post has no idea where any US address is located, it will send US-bound mail to the USPS which will take care of routing the mail to the right location. And just like you could enter pretty much any postal address in a tool such as Google map and get the exact corresponding location, you can query a tool such as the ipstack API with an IP address to get its location.

This has some limitations, though. For instance, you won’t get the exact location from the IP address. What you’ll get is whatever the ISP supplying the IP address can provide. This is often not much more precise than the city or general neighbourhood although some providers return more information than others. Also, if the user is suing a VPN to access your website, what you’d get from such a lookup is the location of the VPN server they’re using.

Main Features Of The ipstack API

The developers of the ipstack API chose to use a modular approach with this product. It consists of various modules, each returning a different type of information. There are, for instance, modules of timezone, currency, or even security. You choose what module to use by using the appropriate parameters when sending the request to the ipstack API.

There are a few advantages to this approach. First, it makes the response easier to handle. If there was only one type of request returning all available data about an IP address, your website’s code would have to parse the whole answer to fetch whatever information it needs. Also, sending smaller requests—with smaller responses—can improve performance.

Let’s have a deeper look at each of the available module, how to call them and what their typical response looks like. That will give you a rather clear overview of what to expect from this powerful tool.

The Location Module

ipstack API - Location Module

The ipstack API’s most basic module is the Location Module. It returns details about the physical location of the IP address provided in the query. You will get the IP address’ country, its main or official language as well as an emoji for the country’s flag which you can use to personalize your user’s experience.

Here’s a typical response to a query to the ipstack API. The first part is included with any query, regardless of the module requested:

ip: "142.243.254.224"
hostname: "142.243.254.224"
type: "ipv4"
continent_code: "NA"
continent_name: "North America"
country_code: "CA"
country_name: "Canada"
region_code: "QC"
region_name: "Quebec"
city: "Montreal"
zip: H2V
latitude: 45.5178
longitude: -73.6046

Most of the returned attributes are self-explanatory but you have detailed documentation on ipstack’s website should you need further help.

But since we’re talking about the location module, let’s have a look at what extra information is returned when using it:

location: Object {}
   geoname_id: null
   capital: "Ottawa"
   languages: Object {}
   code: "en"
   name: "English"
   native: ""English""
   code: "fr"
   name: "French"
   native: ""Français""
   country_flag: "http://assets.ipstack.com/flags/ca.svg"
   country_flag_emoji: "??"
   country_flag_emoji_unicode: "U+1F1E8 U+1F1E6"
   calling_code: "1"
   is_eu: false

As you can see, the extra information you get is mainly the country’s capital, the various languages in use in that location, a link to an image of the country’s flag and the unicode of the country’s emoji.

The Currency Module

ipstack API - Currency Module

The Currency module can be used to further improve your website user experience. This is particularly true of a shopping site as it will allow you to display prices in your user’s currency. The details provided include the local currency name and symbol as well as its plural name. The API won’t convert prices to the proper currency, of course, but it will allow you to use another API to convert your prices into the right currency for your visitors. Let’s have a look at what the output from the currency module looks like:

currency: Object{}
   code: "CAD"
   name: "Canadian Dollar"
   plural: "Canadian dollars"
   symbol: "CA$"
   symbol_native: "$"

The Security Module

ipstack API - Security Module

Probably not as popular—or useful—as the previous modules, the Security Module is still interesting. It returns information about several security aspects of the incoming connection. For instance, it will tell you whether the site is accessed through a proxy and, if so, the type of proxy or whether the request came in through the tor network. It can also tell you that a request is from a crawler rather than an actual user. This type of information can be used to prevent scraping. The module also returns the threat level associated with the IP address provided, potentially allowing for crude protection against malicious requests. Here’s a look at the typical response from the Security Module:

security: Object{}
   is_proxy: false
   proxy_type: null
   is_crawler: false
   crawler_name: null
   crawler_type: null
   is_tor: false
   threat_level: "low"
   threat_types: null

The Time Zone Module

ipstack API - Time Zone Module

Another module which can come in handy to personalize your website visitors user experience is the Time Zone module. It returns some extra details about—you guessed it—your website visitors. One of the places where you could use such a feature in sites that deal with time-sensitive information. It allows you to display the correct time based on your users’ location.

The module provides information like GMT offset, current user time, time code, and time zone location as you can see from this sample response:

time_zone: Object{}
   id: "America/Montreal"
   current_time: "2019-07-09T12:39:22-04:00"
   gmt_offset: -14400
   code: "EDT"
   is_daylight_saving: true

The Connection Module

ipstack API - Connection Module

Last but not least is the Connection Module. This one is rather simplistic. All it returns is the autonomous system (AS) number associated with the source IP address as well as the ISP name. Well, it’s actually the name of the AS owner rather the ISP. Some organizations register their own autonomous system, as is the case in the example below:

connection: Object{}
   asn: 395399
   isp: "City of Montreal"

This is probably the least useful of all the modules but some will find it useful for various analytic purposes.

Working With the ipstack API

We’ve had a look at what information can be returned by the ipstack API, now is the time to have a look at how it is called. The ipstack API offers three different types of queries each with different parameters. There’s the Standard Lookup, the Bulk Lookup and the Requester Lookup. Let’s see how they differ. This is a nice feature as, for instance, the bulk request allows one to combine several requests into one, thereby reducing the total overhead and allowing for faster results. Let’s have a deeper look at each type of request.

Standard Lookup

As you might have guessed, the Standard Lookup is the most basic one. It returns the details about a single IP address that you provide when calling the ipstack API. It’s pretty simple to use and it will work with either IPv4 or IPv6 addresses. To make matters even easier, the product’s documentation provides sample code for the standard request in both PHP (cURL) and JavaScript (Jquery.ajax). If your website is build using either of these, using the API is really a no-brainer. Here’s a sample of a typical call to the API using simple HTML code:

https://api.ipstack.com/123.123.123.123?access_key=YOUR_ACCESS_KEY

In this example, YOUR_ACCESS_KEY would be replaced by the actual access key you got when you signed up for the service. You would also include the actual IP address you want information about rather than the dummy one in this example.

The results for a standard lookup include all the modules by default but extra parameters allow you to tune the request to your exact needs. All the details on the available parameters are clearly explained in the product documentation. Likewise, the default response is a JSON object but parameters can change that to an XML format.

Bulk And Requester Lookups

The Bulk lookup lets you request data against multiple IP addresses within a single request. Just like the Standard Lookup, you can get data on both IPv4 and IPv6 addresses. You can even mix both types of addresses in a single request. The format of the request is the same as that of Standard Lookup except that you specify multiple IP addresses separated by commas.

https://api.ipstack.com/123.123.123.123,124.124.124.124,125.125.125.125,
126.126.126.126?access_key=YOUR_ACCESS_KEY

The last type of request offered by the ipstack API is the Requester Lookup. It will return information on the IP address from which the request is coming. It’s the type of request you’d run from withing a client-side script. To make a Requester Lookup call, all you do is use the keyword “check” in lieu of an IP address. The other available request parameters are the same as for other request types.

https://api.ipstack.com/check?access_key=YOUR_ACCESS_KEY

How Does The ipstack API Perform?

The ipstack API is, more than likely, something you’d integrate into a real-time website. For example, you might want to serve different content based on the client’s location. In order to accomplish that, you need the API to perform blazingly fast. Any delay in the API response will translate into delays displaying the page to your client, something you’d more than likely want to avoid. The same is true no matter what you use the location data for. This is why the response time of the tool you’re using is of the utmost importance. While a geolocation API can provide a much-needed functionality, its failure to perform can have a disastrous impact.

For these reasons, you need to choose an API with good response time. We’ve done much of the hard work of testing various scenarios and ran tests with single or multiple IP addresses, some with more optional parameters, some with less. We were quite pleased with the results and got a rather consistent response time of around 100 ms, regardless of the request. Response was also consistent over time with no apparent “rush hours” when performance would suffer and it was also similar no matter where we launched the requests from. It seems like the ipstack servers are built to handle the load they get. Overall, it seems like using the ipstack API had barely any effect on website response seen from a user’s perspective.

Product Documentation And Support

It is often said that a product is only as good as its documentation. If that is true, the ipstack API has nothing to fear as its documentation is easy to find, use, and understand. The quality of the documentation also gives a measure of how much a software publisher cares about its users. Here again, there is nothing to worry about.

ipstack API - Documentation Home

One of the best thing about this product’s documentation is, oddly enough, how little there is. Everything fits on a single, well laid out web page where you’ll easily find all you need. And to make thing even easier, a menu pane makes accessing any given section a breeze. The documentation covers everything from using the various types of requests to customizing the response.

In the highly unlikely case when you wouldn’t find what you need in the documentation, support is also available. It can take some time to initially reach the tech support but, once you do, they do respond rather quickly.

Pricing

Several pricing plans are available for the ipstack API. At the lowest level, there the free plan. As its name implies, this one is free. It is also somewhat limited. For starters, you can only run up to 10 000 queries per month. It also only includes the Location Module and access to support is limited. It can, however, be a great way to test the product and see how it can integrate into your environment.

The next level is the basic plan. This one sells for $9.99/month or $95.90/year (a 20% discount) and brings the requests limit up to 50 000/month. It also lets you use the Currency, Time Zone and Connection modules. Furthermore, enhanced support is also included with the plan as well as enhanced security through the use of SSL encryption.

The most popular plan is the professional one which sells for $49.99/month or $479.90/year. It gives you everything that’s included in the basic plan but raises the monthly requests limit to 500 000 and also includes the ability to do bulk requests.

At the top is the professional plus plan. It costs $99.99/month or $959.90/year. It is identical to the professional plan but it allows up to 2 million requests per month and it is the only plan to include the security module.

If that ain’t enough, there is also a customizable enterprise plan that can be arranged by contacting ipstack. It can include as many monthly requests as you might need as well various custom solutions based on a client’s specific needs.

In Conclusion

The ipstack API is an excellent product that will let you grab as much of the data you can get from an IP address quickly and easily. It has amazing response time, excellent documentation and good support which combine to make this a great value for the money. Also, the very fact that several Internet giants use it is probably a good testament to the product’s value.

This is definitely a product I’d recommend if you’re looking at exploiting the data that can be gathered for your website visitor’s IP address. And since a free—albeit limited—plan is available, there is no reason why you shouldn’t at least give it a try is see for yourself what it can do for you.

Read Free website visitors geolocation with the ipstack API (Review) by Renaud Larue-Langlois on AddictiveTips – Tech tips to make you smarter