======= Review 1 ======= *** Comments to the author: Summarize the strengths and weaknesses of the paper. Provide a rationale for your rating, and suggested improvements (if appropriate). The topic is very interesting. Thank you for submitting your paper for this conference. In Introduction, I suggest more explanation for phrases and abbreviations, specially Aloha protocol applied to Sigfox and LoRa technologies. I recommend changing the section II, related works into more problem statement section explaining challenges and why this performance evaluation is unique and useful, and the current information as related works can be explained as comparison of this work with previous works. In 3sr paragraph of section III, "Unlike previous studies, which mostly focus on the throughput as an evaluation metric, we use metrics more relevant for the IoT context: the packet success probability and the time each node spends in an ON state (receiving, transmitting or listening to the channel).", this is more comparison of current work than simulation methodology. I suggest simulation set ups for title of this section explaining parameters and assumptions. section IV has three subsections as parameters for comparison, I suggest to introduce these parameters in Introduction section. If possible, I advise to make your results more understandable by adding some algorithm block diagram, flowchart, or mathematical formulas, or algebra or logic statements, prior to your simulation results. Many places in the paper, word "we" is used, I recommend revision and improvement on English. Googd luck. *** Familiarity: Rate your familiarity with the topic of the paper. Very limited expertise (4) *** Relevance to the track and timeliness: Rate the importance and timeliness of the topic addressed in the paper within its area of research. Good (2) *** Technical content and scientific rigour: Rate the technical content of the paper, its scientific rigour and novelty. Solid work of notable importance. (2) *** Quality of presentation: Rate the paper organization, the clearness of text and figures, the completeness and accuracy of references. Substantial revision work is needed. (4) *** Overall evaluation: Please judge whether the paper should be accepted or rejected Accept (1) ======= Review 2 ======= *** Comments to the author: Summarize the strengths and weaknesses of the paper. Provide a rationale for your rating, and suggested improvements (if appropriate). The paper "reboots" some quite old research work about channel access. Authors compare Aloha and CSMA/CA when used to provide channel access to a huge number of nodes that transmit seldom, a typical IOT scenario. They also explore how performance changes when acknowledgments are replaced by preemptive retransmissions. The paper is well written and easy to read and results have some significance. I found the abstract the major weakness of the paper: maybe because the topic has been explored for decades, but the abstract is not really inviting readers to keep on reading, in particular the last part where authors stress about the fact that "the best protocol depends on many parameters [cut] as well as on the metric of interest" which is rather well known. I suggest authors to make the abstract more appealing. In addition I would have preferred authors to show some match about well known results for both CSMA/CA and Aloha and what they present. I.e., for the case of Aloha it could be useful to explore regions of saturation and check the maximum achievable delivery rate is in agreement with the models with infinite nodes. Incidentally this is a configuration very close to that explored by the paper (up to thousands of nodes is like to have infinite number of nodes). Finally I do not agree about the fact authors use metrics that are more relevant for IoT context like the packet success probability rather than throughput: are not these to metrics strictly connected? Some other comments: - I found unusual the choice of AdHocWifiMac: as there is a single AP, authors should have chosen ApWifiMac + StaWifiMac. - It is not clear the timeout chosen for the ack frame in Table I. 75ms seems too much for the chosen datarate (6Mbps). Are authors considering the standard .11 ack reply mechanism? - Maybe authors could cite the GCR mechanism (Unsolicited Retry) introduced by 802.11aa, they are really similar to the preemptive frame retransmission mechanism introduced by authors. *** Familiarity: Rate your familiarity with the topic of the paper. Familiar with this area of research (3) *** Relevance to the track and timeliness: Rate the importance and timeliness of the topic addressed in the paper within its area of research. Acceptable (3) *** Technical content and scientific rigour: Rate the technical content of the paper, its scientific rigour and novelty. Valid work but limited contribution. (3) *** Quality of presentation: Rate the paper organization, the clearness of text and figures, the completeness and accuracy of references. Well written. (2) *** Overall evaluation: Please judge whether the paper should be accepted or rejected Weak accept (2) ======= Review 3 ======= *** Comments to the author: Summarize the strengths and weaknesses of the paper. Provide a rationale for your rating, and suggested improvements (if appropriate). The paper addresses an interesting topic which is the choice of the channel access method for IoT networks based on network characteristics (e.g., number of nodes, energy harvesting, ...). The authors propose no-ack derived schemes and provide their evaluation. The paper built on a solid simulation-based methodology through ns-3 based simulations. *** Familiarity: Rate your familiarity with the topic of the paper. Familiar with this area of research (3) *** Relevance to the track and timeliness: Rate the importance and timeliness of the topic addressed in the paper within its area of research. Good (2) *** Technical content and scientific rigour: Rate the technical content of the paper, its scientific rigour and novelty. Solid work of notable importance. (2) *** Quality of presentation: Rate the paper organization, the clearness of text and figures, the completeness and accuracy of references. Well written. (2) *** Overall evaluation: Please judge whether the paper should be accepted or rejected Accept (1) ======= Review 4 ======= *** Comments to the author: Summarize the strengths and weaknesses of the paper. Provide a rationale for your rating, and suggested improvements (if appropriate). The paper presents a simulation study comparing ALOHA and CSMA/CA, focusing on the success rate. One variation has ACKs removed from the CSMA/CA process. The results are generally interesting. Possible improvements: - Sigfox S appears to be 200 ms. Please add it somewhere near Top definition, to make T_op clear. - Is the Aloha CCA set to infinity or to 0? (bottom of page 2). - If I understand, the maximum number of successful packets per period = 1 / T_op. This seems worth mentioning and worth marking on the graphs (~600 and ~6000, I think). - Capitalise CSMA in Fig. 3, 5 and 6 legends. - Listening and Tx-ing use different power, so Fig. 2 could be misleading. It would be good to indicate approximately how different they are (e.g., same order of magnitude) (if power for energy detect ~= energy Tx) Fig. 2 is good. - I think something is logically wrong with the sentence: “Aloha is also obtaining the best energy consumption results with the lowest number of retransmissions, as shown in Fig. 3b.” The previous sentence was not about Aloha in general, so ‘also’ seems wrong. Please fix it. - In Fig. 6, for CSMA (K=1), it looks like there is seldom a backoff process, which seems strange; I would have thought the initial CCA would usually find the channel busy and thus lead to a backoff process; which would then make the ON time for (CSMA, K=2) ~= 2* (CSMA, K=1). Please discuss/explain this in the paper. *** Familiarity: Rate your familiarity with the topic of the paper. Working in this area of research (2) *** Relevance to the track and timeliness: Rate the importance and timeliness of the topic addressed in the paper within its area of research. Good (2) *** Technical content and scientific rigour: Rate the technical content of the paper, its scientific rigour and novelty. Solid work of notable importance. (2) *** Quality of presentation: Rate the paper organization, the clearness of text and figures, the completeness and accuracy of references. Well written. (2) *** Overall evaluation: Please judge whether the paper should be accepted or rejected Accept (1) ======= Review 5 ======= *** Comments to the author: Summarize the strengths and weaknesses of the paper. Provide a rationale for your rating, and suggested improvements (if appropriate). Motivated by IoT scenarios, authors present a comparison of Aloha and CSMA, and then the performance analysis of some variations. This is hardly a new research problems, as authors themselves acknowledge. Instead of analysis, authors rely on simulations to evaluate the performance in terms of delivery ratios and active times for a number of scenarios. I'm not entirely convinced about the methodology, including the use of the term "transmission opportunity" (why not a normalized arrival rate). Also, the number of scenarios considered is relatively small. The fact that for many figures the results are flat starting from some threshold on the number of nodes suggests that the system is already saturated at this point. *** Familiarity: Rate your familiarity with the topic of the paper. Working in this area of research (2) *** Relevance to the track and timeliness: Rate the importance and timeliness of the topic addressed in the paper within its area of research. Acceptable (3) *** Technical content and scientific rigour: Rate the technical content of the paper, its scientific rigour and novelty. Marginal work and simple contribution. (4) *** Quality of presentation: Rate the paper organization, the clearness of text and figures, the completeness and accuracy of references. Well written. (2) *** Overall evaluation: Please judge whether the paper should be accepted or rejected Borderline paper (3) *********************************************************************************************************************************************************************************************** Comments for previous version submitted at CCNC 2019 *********************************************************************************************************************************************************************************************** ======= Review 1 ======= *** Overall Rating: What is your overall recommendation for the paper? Marginal (30-50%) (2) *** Relevance: How relevant is the topic of this paper to CCNC? Relevant (3) *** Novelty: How novel is the approach/topic presented in the paper? Some novel component (2) *** Technical Strength: Is the paper technically sound and correct Technical solid (3) *** Presentation: What is the quality of presentation of the paper? The paper is overall well written. There are small flaws that are correctable. (3) *** Reviewer Confidence: How comfortable are you in the review decision? Very comfortable and know much of the related work (my area) (3) *** Short Summary: Please give a short summary of the paper The paper presents a performance evaluation of two well-known MAC protocols used in dedicated IoT networks: CSMA and Aloha. The authors investigate the performance of the adapted version of Aloha i.e. removing the provision of ACK transmission. The authors then investigate the performance of CSMA by applying the same concept. It is demonstrated through simulations that the adapted version of Aloha and CSMA yield better performance in terms of packet success probability and energy-consumption. Finally, the authors conclude that a suitable MAC protocol can be chosen for a dedicated IoT network based on the network state and the number of competing IoT nodes. *** Main Strengths (Reasons to Accept): What are the main arguments to accept the paper? The simulation outcomes are the important contributions of the paper as they lead to the conclusion that the best MAC protocol depends on several factors such as packet arrival rate and number of competing IoT nodes. In addition, the comparison of energy-consumption of Aloha and CSMA protocol is another important contribution as the IoT network contains battery-constrained nodes. *** Main Weaknesses (Reasons to reject): What are the main arguments to reject the paper? 1. The authors should have discussed the slotted approach used in Aloha protocol and the retransmission mechanism. 2. The modification to CSMA protocol for IoT networks is not explored completely. It would have been interesting to see the comparison of classical Distributed Coordinated Functions(DCF) and the modified DCF used in IEEE 802.11ah. *** Detailed Comments: Please input additional detailed comments below. ======= Review 2 ======= *** Overall Rating: What is your overall recommendation for the paper? Marginal (30-50%) (2) *** Relevance: How relevant is the topic of this paper to CCNC? Relevant (3) *** Novelty: How novel is the approach/topic presented in the paper? Minor tweak of prior schemes (1) *** Technical Strength: Is the paper technically sound and correct Minor flaws but conclusions are still believable (2) *** Presentation: What is the quality of presentation of the paper? The paper is overall well written. There are small flaws that are correctable. (3) *** Reviewer Confidence: How comfortable are you in the review decision? Comfortable reviewing the paper (close to my area) (2) *** Short Summary: Please give a short summary of the paper The paper compares the performance of Aloha and CSMA in the context of IoT, especially when a backwards channel is not available. *** Main Strengths (Reasons to Accept): What are the main arguments to accept the paper? Extensive simulations comparing popular access schemes. *** Main Weaknesses (Reasons to reject): What are the main arguments to reject the paper? The schemes studied in the paper are well known and have been studied for a long time. The paper is mostly an application of well known concepts. *** Detailed Comments: Please input additional detailed comments below. There are many solid analytical models for Aloha and CSMA, the paper could compare analytical results and simulations. ======= Review 3 ======= *** Overall Rating: What is your overall recommendation for the paper? Reject (50-75%) (1) *** Relevance: How relevant is the topic of this paper to CCNC? Relevant (3) *** Novelty: How novel is the approach/topic presented in the paper? Minor tweak of prior schemes (1) *** Technical Strength: Is the paper technically sound and correct Significant flaws that may call key result of the paper into question (1) *** Presentation: What is the quality of presentation of the paper? The paper can benefit from significant revision (2) *** Reviewer Confidence: How comfortable are you in the review decision? Somewhat comfortable reviewing (but not my area) (1) *** Short Summary: Please give a short summary of the paper Authors address the problem of sharing the communication medium between connected nodes and base stations in IoT networks. *** Main Strengths (Reasons to Accept): What are the main arguments to accept the paper? The paper contribution includes comparing two channel access methods (e.g., CSMA and Aloha) in IoT networks through a simulation study and study their characteristics/benefits for traffic arrival rate, connected objects number, energy consumption, etc. *** Main Weaknesses (Reasons to reject): What are the main arguments to reject the paper? Paper novelty is on the low side and does not present any concrete abstractions of IoT networks to discuss new challenges in channel access. It does not make a strong case of why CSMA and Aloha need to be revisited in studies in a different perspective for IoT networks, perhaps more clear differentiation from related works would be more convincing. *** Detailed Comments: Please input additional detailed comments below. *********************************************************************************************************************************************************************************************** Comments for previous version submitted at VTC Fall 2018 *********************************************************************************************************************************************************************************************** ======= Review 1 ======= Relevance 4: Good match Novelty 3: Incremental Contribution 2: Minor contribution Presentation 3: Adequate Recommendation 2: Reject Comments The motivations and contributions of this paper are not convincing enough. Although the paper has abundant simulation results, I did not see any mathematical model to verify the simulation results. Moreover, no performance study with the existing methods. ======= Review 2 ======= Relevance 5: Perfect match Novelty 1: Been done Contribution 1: Little or no contribution Presentation 4: Clear Recommendation 2: Reject Comments It is okay in my view to conduct a simulation based performance evaluation in a novel setting. The present paper could do better in such a venture, if certain key ideas were not conflated and muddled. Contrary to the authors' claims (in the last paragraph of Section II), slotted Aloha is not an assumption. Having time slotted simply doubles the channel efficiency. There are quite a few issues of this nature throughout the paper. Compared to what is presented, the authors have room to carry out more meaningful performance evaluation pertaining to the key ideas around such random access schemes as pure/slotted Aloha and CSMA. On more than a few occasions, there have been claims and statements that are not quite consistent with long established and tested studies and wisdom (see summarized exposition in, e.g., Chapter 4 of Data Networks (2nd Ed) by Dimitri Bertsekas and Robert Gallager, as well as [6], among others). After all, how pure/slotted Aloha and CSMA fare in performance are reasonably well-understood. The saturated and non-saturated contrast is not as consequential as the authors seem to think. If you have a traffic model of unsaturated network where arrival is homogeneous among all nodes and with low variance, then you have a mean number of busy/contending nodes at any given time and a saturated model with that many nodes would give you expected performance that is equivalent on aggregate with the aforementioned unsaturated network (by the way, what is the authors traffic model? A mean packet inter-arrival of 0.1s or 1s doesn't say whether the arrivals are Poisson or uniform or something else; also the buffering assumptions are unstated or sketchy.) If the authors simulate in a doubly stochastic setting, where the rate parameter of the arrival statistic is also taken as a random number rather than as a constant and with nodes being non-homogeneous, that would probably be something. In a single hop network, the normalized measure of the MAC layer random access throughput (at packet level) has the straightforward interpretation as channel utilization or efficiency. Importantly, this has a one-to-one correspondence with packet success probability, and you'd have to factor in the packet size (measured in length of time) to do that (what is the packet size in the authors' evaluation, by the way?). Even the mean packet delay in a single hop network is directly derivable from Little's law once you have the throughput. Packet delivery/reception ratio makes sense as a distinctive metric in evaluating multihop routing, and not in this case. So the author's mission to formulate new performance metrics has apparently fallen short from producing anything remarkable. The authors say "MAC layer directly transmits any message produced by the application layer". Aren't there supposed to be at least something like an end-to-end layer (i.e., transport layer) before the application layer even if you don't think of the network layer? Relatedly, the second sentence in the first paragraph of Section V-A is annoyingly cryptic. With a more careful formulation of the study, the authors can potentially salvage valuable findings from their simulation. It would be appreciable on their part to put the modified simulator on the github as well then. ======= Review 3 ======= Relevance 4: Good match In this paper, the authors evaluate the performance of multiple channel access methods for IoT application from the view point of connectivity. This manuscript is well suited to the conference. Novelty 2: Derivative This paper just evaluates the performance of multiple random access channel methods by numerical computation. Thus, the novelty is not so high. Contribution 3: Worthwhile contribution The obtained results may be beneficial to the other researchers when they choose the random access channel method for IoT networks. Presentation 3: Adequate Recommendation 3: Possible accept Comments In Fig. 4, the authors evaluate the average node-to-gateway delay. However, this evaluation may not be appropriate as some of the packets may be dropped. ======= Review 4 ======= Relevance 5: Perfect match Novelty 2: Derivative Contribution 2: Minor contribution Presentation 4: Clear Recommendation 2: Reject Comments 1. The three methods are quite old, and there have been well and widely addressed before. So I don’t understand why the authors still focus on these. Therefore, the motivation should be clear shown, and the detailed comparisons should be provided to make readers to know the contributions of this work. 2. The maximum value of abscissa is displayed incorrectly In Figures 1 and 2. 3. In page 3 part V. SIMULATION RESULTS, "In the first region, for small networks up to 100 nodes, the CSMA and Aloha success probabilities are both equal to 1, while the Aloha No Ack probability starts at 1 and slowly decreases.” However, the packet success probability of the three protocols shown in Fig. 1(a) should be close to 1, the description is not serious. ======= Review 5 ======= Relevance 4: Good match Novelty 2: Derivative Contribution 3: Worthwhile contribution Presentation 4: Clear Recommendation 2: Reject Comments the contributions of this paper are insufficient for being accepted