From 7e6f73a6cab03c8c6c2013c97ff768f05b11fd42 Mon Sep 17 00:00:00 2001 From: Matt Brown Date: Fri, 21 Feb 2014 14:15:50 -0500 Subject: [PATCH 1/4] WIP on logging aggregator information --- .../content/pages/web-framework-logging.rst | 53 +++++++++++++++++++ source/content/pages/web-frameworks.rst | 28 ---------- 2 files changed, 53 insertions(+), 28 deletions(-) create mode 100644 source/content/pages/web-framework-logging.rst diff --git a/source/content/pages/web-framework-logging.rst b/source/content/pages/web-framework-logging.rst new file mode 100644 index 000000000..b64900394 --- /dev/null +++ b/source/content/pages/web-framework-logging.rst @@ -0,0 +1,53 @@ +--------------------- +Web Framework Logging +--------------------- +Logging is a common mechanism for monitoring web applications written with a +web framework. Runtime exceptions that prevent code from running are +important to log to investigate and fix the source of the problems. +Informational and debugging logging also helps to understand how the +application is performing even if code is working as intended. + +Logging is often grouped into several categories: + +1. Information +2. Debug +3. Warning +4. Error + +Logging errors that occur while a web framework is running is crucial to +understanding how your application is performing. +`Raven `_ is a Python client for the +`Sentry `_ exception logging and +aggregation application. Raven provides the way to send exceptions to +Sentry, which should be deployed on a separate server from your production +infrastructure. Raven can also be used by Python scripts to send other +log data to Sentry for aggregation. Sentry provides a clean web application +interface for viewing the exceptions. Sentry can also be configured with a +mail plugin to send emails when exceptions occur. + + +------------------- +Logging Aggregators +------------------- +When you are running your application on several servers, it is helpful +to have a monitoring tool called a "logging aggregator." You can configure your +application to forward your application and system logs to one place that +allows you to view, search, and monitor logging events across all of your machinces. + +Another advantage of log aggregators tools is they allow you to set up custom alerts +and alarms so you can get notified when error rates breach a certain threshold + +Log Aggregator Third Party Services +----------------------------------- +`loggly `_ Loggly is a third party cloud based application that +aggregates logs. They have instructions for every major language, including python. It includes email +alerting on custom searches. +`splunk `_ Splunk offers third party cloud and self hosted solutions +for event aggregation. It excells at searching and data mining any text based data. + +Open Source Projects +-------------------- +`Logstash `_ +`Graylog2 `_ Provides a central server for log aggregation as well as a GUI for +browsing and searching through log events. There are libraries for most major languages, including python. +Saves data in elasicache. diff --git a/source/content/pages/web-frameworks.rst b/source/content/pages/web-frameworks.rst index 219334982..a500d568a 100644 --- a/source/content/pages/web-frameworks.rst +++ b/source/content/pages/web-frameworks.rst @@ -150,34 +150,6 @@ The tuts+ `Flask tutorial `_ is a Python client for the -`Sentry `_ exception logging and -aggregation application. Raven provides the way to send exceptions to -Sentry, which should be deployed on a separate server from your production -infrastructure. Raven can also be used by Python scripts to send other -log data to Sentry for aggregation. Sentry provides a clean web application -interface for viewing the exceptions. Sentry can also be configured with a -mail plugin to send emails when exceptions occur. - - Web Framework Resources ======================= `Bottle `_, From 9a7573c0dc3042c17dae175279352261c0061689 Mon Sep 17 00:00:00 2001 From: Matt Brown Date: Sun, 9 Mar 2014 21:41:38 -0400 Subject: [PATCH 2/4] Finish up the writeup on logging aggregators --- .../content/pages/web-framework-logging.rst | 21 ++++++++++++------- 1 file changed, 13 insertions(+), 8 deletions(-) diff --git a/source/content/pages/web-framework-logging.rst b/source/content/pages/web-framework-logging.rst index b64900394..2303b7251 100644 --- a/source/content/pages/web-framework-logging.rst +++ b/source/content/pages/web-framework-logging.rst @@ -25,29 +25,34 @@ log data to Sentry for aggregation. Sentry provides a clean web application interface for viewing the exceptions. Sentry can also be configured with a mail plugin to send emails when exceptions occur. - ------------------- Logging Aggregators ------------------- When you are running your application on several servers, it is helpful -to have a monitoring tool called a "logging aggregator." You can configure your -application to forward your application and system logs to one place that -allows you to view, search, and monitor logging events across all of your machinces. +to have a monitoring tool called a "logging aggregator". You can configure your +application to forward your system and application logs to one location that +provides tools such as to viewing searching, and monitoring logging events across your cluster. -Another advantage of log aggregators tools is they allow you to set up custom alerts -and alarms so you can get notified when error rates breach a certain threshold +Another advantage of log aggregatortion tools is they allow you to set up custom alerts +and alarms so you can get notified when error rates breach a certain threshold. Log Aggregator Third Party Services ----------------------------------- `loggly `_ Loggly is a third party cloud based application that aggregates logs. They have instructions for every major language, including python. It includes email alerting on custom searches. +`papertrail `_ Paper trail is similar to both loggly and splunk and provides +integration with S3 for `splunk `_ Splunk offers third party cloud and self hosted solutions -for event aggregation. It excells at searching and data mining any text based data. +for event aggregation. It excells at searching and data mining any text based data. Open Source Projects -------------------- -`Logstash `_ `Graylog2 `_ Provides a central server for log aggregation as well as a GUI for browsing and searching through log events. There are libraries for most major languages, including python. Saves data in elasicache. +`Logstash `_ Similar to Graylog2, logstash offers features to programatically +configure log data workflows. +`Scribe `_ A project written by facebook to aggregate logs. It's designed +to run on multiple servers and scale with the rest of your cluster. Uses the Thrift messagaing format so it can +be used with any language. From d30e8c8c55929dd04076ce9693dc4abb0d184cb6 Mon Sep 17 00:00:00 2001 From: Matt Brown Date: Sun, 9 Mar 2014 22:04:05 -0400 Subject: [PATCH 3/4] Move content to new logging section and update formatting --- .../pages/19-logging/1901-logging.markdown | 27 +++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/source/content/pages/19-logging/1901-logging.markdown b/source/content/pages/19-logging/1901-logging.markdown index 028a44364..68d1b3789 100644 --- a/source/content/pages/19-logging/1901-logging.markdown +++ b/source/content/pages/19-logging/1901-logging.markdown @@ -28,4 +28,31 @@ log data to Sentry for aggregation. Sentry provides a clean web application interface for viewing the exceptions. Sentry can also be configured with a mail plugin to send emails when exceptions occur. +## Logging Aggregators +When you are running your application on several servers, it is helpful +to have a monitoring tool called a "logging aggregator". You can configure your +application to forward your system and application logs to one location that +provides tools such as to viewing searching, and monitoring logging events across your cluster. + +Another advantage of log aggregatortion tools is they allow you to set up custom alerts +and alarms so you can get notified when error rates breach a certain threshold. + +### Log Aggregator Third Party Services +* [loggly](https://www.loggly.com/) Loggly is a third party cloud based application that +aggregates logs. They have instructions for every major language, including python. It includes email +alerting on custom searches. +* [papertrail](https://papertrailapp.com/) Paper trail is similar to both loggly and splunk and provides +integration with S3 for +* [splunk](http://www.splunk.com/) Splunk offers third party cloud and self hosted solutions +for event aggregation. It excells at searching and data mining any text based data. + +###Open Source Projects +* [Graylog2](http://graylog2.org/) Provides a central server for log aggregation as well as a GUI for +browsing and searching through log events. There are libraries for most major languages, including python. +Saves data in elasicache. +* [Logstash](http://logstash.net/) Similar to Graylog2, logstash offers features to programatically +configure log data workflows. +* [Scribe](https://github.com/facebook/scribe) A project written by facebook to aggregate logs. It's designed +to run on multiple servers and scale with the rest of your cluster. Uses the Thrift messagaing format so it can +be used with any language. From 56fd8af37198e175695c8f942c36b0c0174b12bb Mon Sep 17 00:00:00 2001 From: Matt Brown Date: Sun, 9 Mar 2014 22:04:46 -0400 Subject: [PATCH 4/4] Remove personal page added to store logging framework information --- .../content/pages/web-framework-logging.rst | 58 ------------------- 1 file changed, 58 deletions(-) delete mode 100644 source/content/pages/web-framework-logging.rst diff --git a/source/content/pages/web-framework-logging.rst b/source/content/pages/web-framework-logging.rst deleted file mode 100644 index 2303b7251..000000000 --- a/source/content/pages/web-framework-logging.rst +++ /dev/null @@ -1,58 +0,0 @@ ---------------------- -Web Framework Logging ---------------------- -Logging is a common mechanism for monitoring web applications written with a -web framework. Runtime exceptions that prevent code from running are -important to log to investigate and fix the source of the problems. -Informational and debugging logging also helps to understand how the -application is performing even if code is working as intended. - -Logging is often grouped into several categories: - -1. Information -2. Debug -3. Warning -4. Error - -Logging errors that occur while a web framework is running is crucial to -understanding how your application is performing. -`Raven `_ is a Python client for the -`Sentry `_ exception logging and -aggregation application. Raven provides the way to send exceptions to -Sentry, which should be deployed on a separate server from your production -infrastructure. Raven can also be used by Python scripts to send other -log data to Sentry for aggregation. Sentry provides a clean web application -interface for viewing the exceptions. Sentry can also be configured with a -mail plugin to send emails when exceptions occur. - -------------------- -Logging Aggregators -------------------- -When you are running your application on several servers, it is helpful -to have a monitoring tool called a "logging aggregator". You can configure your -application to forward your system and application logs to one location that -provides tools such as to viewing searching, and monitoring logging events across your cluster. - -Another advantage of log aggregatortion tools is they allow you to set up custom alerts -and alarms so you can get notified when error rates breach a certain threshold. - -Log Aggregator Third Party Services ------------------------------------ -`loggly `_ Loggly is a third party cloud based application that -aggregates logs. They have instructions for every major language, including python. It includes email -alerting on custom searches. -`papertrail `_ Paper trail is similar to both loggly and splunk and provides -integration with S3 for -`splunk `_ Splunk offers third party cloud and self hosted solutions -for event aggregation. It excells at searching and data mining any text based data. - -Open Source Projects --------------------- -`Graylog2 `_ Provides a central server for log aggregation as well as a GUI for -browsing and searching through log events. There are libraries for most major languages, including python. -Saves data in elasicache. -`Logstash `_ Similar to Graylog2, logstash offers features to programatically -configure log data workflows. -`Scribe `_ A project written by facebook to aggregate logs. It's designed -to run on multiple servers and scale with the rest of your cluster. Uses the Thrift messagaing format so it can -be used with any language.